Can't pip install private Github repo with bash script - bash

It's been a while since I asked this question. To simplify, I just want a lifecycle configuration in AWS SageMaker which can successfully install a private GitHub repo.
I'm trying to install a private github repo with a bash script. The script does the following:
makes sure there's an ssh agent active
adds the ssh key from a persistent portion of memory
attempts to install the github repo
This is all happening in a SageMaker AWS EC2 instance via a lifecycle configuration. The implementation looks something like this:
HOME=/home/ec2-user/
ENVPIP=$HOME/anaconda3/envs/tensorflow2_p36/bin/pip
eval "$(ssh-agent -s)"
ssh-add ${HOME}SageMaker/Setup/id_rsa
yes | $ENVPIP install git+ssh://git#github.com/...
Running this, I get the following error:
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****#github.com/...' /tmp/pip-req-build-ysacff_l Check the logs for full command output.
Here's all the pertinent output from cloudwatch:
Agent pid 5146
Identity added: /home/ec2-user/SageMaker/Setup/id_rsa (/home/ec2user/SageMaker/Setup/id_rsa)
2020-09-07T17:11:00.605-04:00
Collecting git+ssh://****#github.com/********1/*****-*****Library
Cloning ssh://****#github.com/********1/*****-*****Library to /tmp/pip-req-build-ysacff_l
2020-09-07T17:11:00.605-04:00
Copy
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****#github.com/********1/*****-*****Library' /tmp/pip-req-build-ysacff_l Check the logs for full command output.
looking into it, this seems like an issue with the cloning protocol, but I couldn't find anything pertinent to ssh.
P.s.
running the same few lines in the terminal works
I sanity checked the url to the repo, went right to it, so I don't think its a problem with anything after the ...
Updates:
tried updating git with yum install git. Apparently my version is up to date, so doing this resulted in the same error.
I commented out the pip install so that the EC2 Instance would start up successfully, then ran curl http://www.google.com, which resulted in a bunch of html. So it appears, at least after the EC2 instance boots, outbound traffic is allowed.
running curl http://www.google.com within the bash script (lifecycle configuration, with the problematic code commented out) results in the same html output, and the instance started up perfectly. this leads me to believe that there is, indeed, outbound traffic allowed on instance startup
a lot of people have viewed this question, and no one has answered it. I'm not married to the specific way I'm trying to install the repo, so if there are any working alternatives I'll gladly take them.
Is it possible that I'm encountering a race condition with some other system? this is happening close to when the instance starts. Are their any way to check that all dependent systems are running?
while doing some other stuff, in console I got the same error. I reinitialized the ssh agent, added the key, and it worked. I wonder if it's a race condition between eval "$(ssh-agent -s)" and yes | $ENVPIP install git+ssh://git#github.com/...?

Related

Use go get to require dependency from private github repo on WSL 2

I'm trying out WSL 2 on windows 10 and it's gone well so far, but I've been struggling for 2 weeks to make this work, because for some reason go get doesn't use or is not able to make the Git Credentials Manager to prompt for my credentials.
I followed this blog to set up WSL2 with GCM https://www.edwardthomson.com/blog/git_credential_manager_with_windows_subsystem_for_linux.html
And it works very good for most of the daily tasks like cloning, read and write. But when using go get I get this error.
go get <remote github repo>#<latest commit id>
go: <remote github repo> 681dceefc81203e094872401c184d038090d6049 => v0.0.17-0.20200501212733-681dceefc812
go get: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: verifying module: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: reading https://sum.golang.org/lookup/<remote github repo>#v0.0.17-0.20200501212733-681dceefc812: 410 Gone
server response:
not found: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/232ff028cb2fdebd254e30bfc612843483f0fe3fbeb18d5fc8fb4b20f21c9021: exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Already tried ssh-keys and the solutions proposed here go get results in 'terminal prompts disabled' error for github private repo
But the error remains the same, when enabling env GIT_TERMINAL_PROMPT=1 nothing happens, I guess it's because WSL 2 doesn't have the permissions to do that. Anyway I also tried this tool https://github.com/microsoft/Git-Credential-Manager-for-Mac-and-Linux and by setting a variable for plain credentials store, it prompts in the terminal for credentials. But I'm using 2FA because it's required by the organization and the prompt only asks for username and password, so the authentication fails.
So I have to reach out to a mate who is using Mac. He is able to go get the dependency to affect go.mod, make a commit and push the change so I can pull it and continue from there. But of course this is not ideal, and he doesn't have any problem, he uses osxkeychain to manage his git credentials.
Anyone has faced this issue? or know how to solve it? Thank you so much in advance.
Go is not able to understand that certain modules are private and their checksum should not be validated against go's checksum library. The following error comes from that
verifying module: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: reading https://sum.golang.org/lookup/<remote github repo>#v0.0.17-0.20200501212733-681dceefc812: 410 Gone
If possible use at least go 1.13 or, higher. Go had introduced an env variables by name GOPRIVATE, GONOPROXY and GONOSUMDB for managing private modules better. Simplest way to signal to Go that you are importing a private repo is to use GOPRIVATE. Set the pattern of private repos to GOPRIVATE env variable to suppress checksum validation and usage of GOPROXY. Example below avoids checksum for all repos in that hierarchy:
GOPRIVATE=github.com/<your org>/*
Check out answers here and here. You can also do go help module-private for help.

Windows, Git and the SSH url/path of a bare repository

I am trying to set up a git repository on our Windows server.
I installed OpenSSH for Windows and Git for Windows.
The OpenSSH server runs on port 2345 to filter out generic bots.
I created a bare and shared repository on C:\git\protocolrepo.git
I couldn't reach it with the URL:
ssh://User#domain#server.com:2345:C:/git/protocol.git
It seems to me to be a malformed URL but everywhere I look on the web I get this, but it just doesn't work.
As I found out in this thread: https://github.com/PowerShell/Win32-OpenSSH/issues/895, the shell might be the cause, so I set the powershell as the default shell of OpenSSH.
This didn't work either and I still needed to use this oddly formed URL.
So I installed Cygwin and set the Cygwin bash as the default shell, and used a more appropriately formed url:
ssh://User#domain#server.com:2345/cygdrive/c/git/protocol.git
I had a little bit more success with this, but it now states the following error message:
fatal: '/cygdrive/c/git/protocol.git' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
The access rights should be fine as I make use of the same user alltogether, but I'm not 100% sure how this works on Windows.
I'm not sure either how Git/ssh handles this path on Windows.
On a Linux based server this worked just fine with /git/protocol.git path.
Pushing a new branch to a bare repository is something I haven't done in a while either so that also might be done wrong here, but I can't recall it to be an issue.
What would you recommend to debug this issue?
I would start by checking if an interactive SSH session works, before considering using a Git repo SSH URL.
ssh -p 2345 User#domain#server.com
If it does, try and access /C/git/protocol.git in that interactive shell.
If that works, then try for your git commands the URL
ssh://User#domain#server.com:2345/C/git/protocol.git
# or
ssh://User#domain#server.com:2345/c/git/protocol.git

Jenkins issue with Git [Permission denied (publickey).] on Windows

I know that thousands of topics touch my problem but I am fighting this from several days with almost zero progress ;(.
I installed Jenkins on Windows machine.
Then acquired Git plugin and the Git itself.
Everything is latest version for the day of my writing.
I was just happy to start to play with Jenkins but I failed at configuring my Bitbucket repository to be used by Job.
I set the exact path to git.exe (as it seemed the reason for initial problems) [C:\Program Files\Git\cmd\git.exe].
Then I tried accessing the repository by HTTPS with account credentials, many strange errors here so I thought let's try with SSH.
I have generated rsa keys, set the public key in the Bitbucket repository. Then I created credential configuration in Jenkins for SSH authentication with passphrase.
I have many problems here so for temporary workaround I explicitly set the value of private key.
Finaly I got to the place where I am now:
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
When I try to manualy copy repository, it politely asks me about passphrase and the copy is done. When Jenkins job does it the above happens. With help of %USERNAME% and %USERDOMAIN% in the Job's script I get the same values like when I do it manually.
After plenty of hours I can say I can't use Git in Jenkins ;(
UPDATE
I don't believe that it could be so simple. As mainframer mentioned, I left passphrase empty for the whole SSH authentication and it really worked (the repository is cloned even that Jenkins still shows the same error in job's configuration page). I only wonder about any consequences of such setup, but working thing is more valuable for me now.
You mean leaving empty passphrase while generating keys and then also no passphrase config in Jenkins ssh credentials?
Yes, as Jenkins wouldn't be able to feed the passphrase to the ssh session.
But make sure the Jenkins slave service runs with your account (and not with the system account). It needs your account to find the ssh keys in %USERPROFILE%\.ssh\id_rsa(.pub)

Need help setting up a git server on windows

i'm trying to set up a git server on Windows, but i'm having some issues getting it all to work.
I have locally created a normal repository, and remotely i created a bare repository. On the local repository i added a single text file and committed it, but when I try to push it into the remote repository I always get the following message:
fatal: protocol error: bad line length character: fata
I searched SO and other sources, and most of them suggest it's an issue regarding command echos. I'm using freeSSHd as a SSH solution (remote repository is hosted on a windows server), and I tried to use both the git bash and the windows CMD as a command shell.
I start CMD with /Q to disable echoing and /K to change directory to a directory where repositories are located, so I don't think that would be a problem.
Using the remote desktop, i can clone the repository to a folder next to it, and using the git bash locally i can access the SSH shell and also clone the repository in the same way. But using git clone ssh://<address>:/myRepo.git I always get the above message (The SSH's working folder is the same where the repository is located). Does anyone have any idea what's going on? How can I see what command is triggering the error, and how can I see the full error message?
I also met the same error using freeSSHd as a ssh solution for git server on Windows. I couldn't find a solution for a whole day and gave up. :(
Later I found another powerful ssh server from Bitvise called WinSSHD worked well. It has free version for personal use. I suggest you to switch to it. Though I'd also like to know if there's a fix to the error we both met.
To setup ssh server with WinSSHD is quite simple, and you can add virtual accounts with private/public key access.
The key part is to setup the ssh access for git server. Please follow the steps of the blog here.
It should work well for Windows git client. For Mac, you may meet an error as follows.
grp.sh: No such file or directory
fatal: Could not read from remote repository.
To fix it, you need to create the two files gup.sh and grp.sh in your git bin directory (GIT_PATH/bin or GIT_PATH/libexec/git-core configured in system environment variable PATH) in your git server.
The content of gup.sh:
git-upload-pack.exe $*
The content of grp.sh:
git-receive-pack.exe $*

Git not working with remote server

I have git installed and working as expected on my local system or a network share, but have not been able to get it to access the remote bare repository on my server using ssh.
I have installed msysgit and CopSSH on the server and msysgit, putty and tortoisegit on my local system. I am able to open a putty session or use plink successfully -- at least I believe plink is successful, but more on that in a bit.
When I attempt to clone a remote repository I get this:
D:\Temp> git clone uname#server:e:/path/test.git Test
Cloning into Test...
fatal: protocol error: bad line length character: Micr
The error message has been consistent for a while, but at times (from one reinstall attempt to another) the last four characters will be 'Micr', 'Allo' or something else I don't remember right now. In the currently installed state it is 'Micr'.
So the big question is what am I not doing right?
Although I imagine it would be much easier to correct what was wrong if I could get the actual error message instead of the generic one followed by the first four characters of the actual error message. Any idea if that is possible?
First, you have variables like export GIT_TRACE=1 and export GIT_TRACE_PACKET=1 which can help debugging this kind of situation.
Second, this error is often caused by any kind of display done during the ssh session (see for instance this thread), like a .bashrc on the server side with an echo in it.
I have a wrapper set-up around ssh that sets the window title of the shell to the ssh command argument. That is what messes things up!
See also Git FAQ
Other causes include:
using the right ssh
correctly forcing to use git-shell
using --exec-path from the server_arg on the server side
Note: with Git 2.6+ (Q3 2015), see also "GIT_TRACE_PACKFILE".

Resources