Problem Statement
I am looking for a recommendation about the proper way to set up SSH keys for connecting to Git on our Jenkins master and slaves. There are other similar questions here about Git, Jenkins, and SSH (the closest being this), but none seems to answer this question with a specific recommendation about the right way to do this.
Background & Testing
I am working on a prototype implementation of Jenkins with on-premise GitLab. We have a few years of experience with Jenkins, but integrating it with Git is new for us. This is also the first time we've configured any slave nodes. At this point, we're just trying to get simple polling to work, no fancy push notifications.
The master Jenkins instance is running on Linux (Red Hat Linux 6). There are currently three slaves: one other Red Hat Linux server, and two servers running Windows Server 2012 R2. The Jenkins slave on the Windows servers runs as a Windows service. The master and all slaves are running up-to-date Git v2.10.0 (64-bit).
We're intending to use SSH to access GitLab. I created an SSH key with a passphrase and saved it on disk on the master. Then, I configured the public key as a deploy key for my test project in GitLab. Finally, I added a new credential in Jenkins.
In my first test project in Jenkins, I configured the GitLab repository URL and selected the Jenkins credentials created above. Then I built the project on the master. Jenkins made the SSH connection to GitLab with no problem and the build succeeded. Then I forced the build over to the Linux slave, with the same results.
Unfortunately, when forcing the build over to either of the Windows slaves, the build doesn't work like it does on Linux.
Errors on Windows
(Edited from my original question, based on further research.)
I initially got this error, which prevented any SSH connections from happening:
stderr: Unable to negotiate with xxxx port 22: no matching host key type found. Their offer: ssh-dss
This has to do with negotiation at the SSH level. The newer SSH client bundled with Git won't use ssh-dss by default, and that's all my relatively old SSH server is capable of offering.
To get past this, I added %HOME%/.ssh/config containing the following lines, as documented by OpenSSH under Legacy Options:
Host xxxx
HostKeyAlgorithms +ssh-dss
This forces the Git SSH client to accept ssh-dss connections.
Around the same time, I was struggling with how to get verbose SSH output out of the Windows SSH client bundled with Git, because the usual suggestion to set GIT_SSH='ssh -vvv' doesn't work on Windows. Based on an Ask Ubuntu question and a similar Stack Overflow question, I ultimately ended up with this in %HOME%/.ssh/config:
Host xxxx
HostKeyAlgorithms +ssh-dss
LogLevel DEBUG3
Having the debug output available gave me a better clue about what is going on. In the end, I can make a build succeed on Windows, but only if the SSH key that I use does not have a passphrase. The hint is here in the debug output:
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,hostbased
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password,hostbased
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\JENKIN~1.TMS\\AppData\\Local\\Temp\\ssh2489775972592020656key
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug2: no passphrase given, try next key
debug2: we did not send a packet, disable method
For some reason, Jenkins is failing to set up the SSH key with the proper passphrase on Windows, where it works just fine on Linux. Unfortunately, Jenkins doesn't dump the debug output from SSH except when the command fails, so I can't see exactly what the Linux build is doing differently.
Questions
The (possibly-naive) way I am trying to do this clearly works on Linux, but not on Windows. Is it supposed to work on Windows?
Is this the correct way to manage SSH keys for my situation, or is there some better way to do it? I was hoping to avoid having to manage key files on disk for every slave as this infrastructure grows.
Assuming this is the correct mechanism to use, can someone point me at instructions or an answer that definitively explains how to get this working? I feel like I must be missing something obvious...?
Related
Recently I had to get my hard drive replaced on my work machine and thus had to reconfigure everything. As a result I had to reinstall git bash. Before I was able to ssh fine into wpengine and now I cannot.
I am able to connect via the regular windows terminal fine but when I try with git bash I am getting the "Permission denied (publickey)." error for the same exact command.
I have tried all the suggested options from wpengine and in the different questions related to this on other SE questions and nothing is working.
I am using a Windows machine on windows 10.
Here are the following things I have tried:
Regenerating the key and adding it to my user public keys again and
then waiting 24 hours.
Adding the config details to the ssh_config file in C:\Program
Files\Git\etc\ssh
Adding a config file to my /User/username/.ssh/ folder.
I have tried using the following link and adding the wpengine rsa file: https://gist.github.com/jherax/979d052ad5759845028e6742d4e2343b as well.
Any and all help would be appreciated.
My guess is there is some kind of permissions issue going on the local machine?
Why would the request from git bash terminal to wpengine look different from windows command terminal?
I did solve my issue. If it helps you please use it!
When I used the command to the ssh host with : ssh -v user#environment.wpengine.ssh.net info
I got back this among the debug errors:
debug1: Offering public key: /c/Users/USERNAME/.ssh/KEY_FILENAME RSA XXXXXXXXXXXXXXXXXXXXX explicit
debug1: send_pubkey_test: no mutual signature algorithm
debug1: No more authentication methods to try.
user#environment.ssh.wpengine.net: Permission denied (publickey).
After finding this page:
https://transang.me/ssh-handshake-is-rejected-with-no-mutual-signature-algorithm-error/
I was able to solve the issue by adding the line:
PubkeyAcceptedAlgorithms +ssh-rsa to my ssh config file.
Honestly I am not even 10% certain on WHY this worked, however, it solved my problem.
How To Connect with SSH In WPEngine
If you are having trouble connecting to SSH in WPEngine Following are the commands which I used:
ssh-keygen -t rsa -b 4096 -f c:/users//.ssh/wpengine_rsa
Add Fingerprint in WPEngine My Profile – SSH
Add Config file
Host *.ssh.wpengine.net
IdentityFile ~/.ssh/wpengine_rsa
IdentitiesOnly yes
Connect with your wordpress website (windows command prompt)
ssh environment#environment.ssh.wpengine.net
I keep most of my personal projects on BitBucket by Atlassian. As a natural choice, I use SourceTree app (their product) as git GUI client on Windows, and I'm happy with it. For projects where ssh git is available, I prefer ssh over https. SourceTree plays very well with projects hosted on BitBucket. Although it offers both of ssh agents: OpenSSH or PuTTY, its default selection is PuTTY/Plink (perhaps because PuTTY is more Windows-familiar).
Recently my establishment requested to host some projects on its own server. At first look it's a git server using GitLab opensource. I can use SourceTree with project hosted here using https just fine, however when it comes to ssh, the only choice of SSH agent is OpenSSH. The only key pair it would use (unless specified in config) is ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa (located in %USERPROFILE%\.ssh\
I've tried to:
Load ~/.ssh/id_rsa into PuTTY Key Generator
Convert it to PuTTY format (.ppk) file
Load the .ppk into Pageant (PuTTY authentication agent).
Despite the key being loaded and kept in the memory by Pageant, the connection with the server failed all the time, e.g. git: fatal: Could not read from remote repository. The only way to make it work is to start ssh-agent and ssh-add (go with OpenSSH).
Since I have Pageant running usually in the background, I find it more convenient to use (e.g. the keyphrase to open the private key is long/complex, and I don't remember it, and it can be copy-pasted from KeePass, while in the case of OpenSSH, cmd console does not allow me to paste it, too bad).
Is there anyway to make the ssh authentication to GitLab done via PuTTY instead of OpenSSH?
Did you connect to the SSH server using PuTTY before using plink? If not a reason could be that plink is refusing to connect to the server, because the ssh hostkey isn't verified yet. Another reason could be that the SSH server requires ciphers which are not supported by PuTTY. You can only find out if you connect with PuTTY with the same version as plink.
Use TortoiseGitPlink (from TortoiseGit) to circumvent this issue, as it will popup a messagebox asking whether to accept the hostkey or not.
I have trouble getting the official Windows ADT 21.1.0 distribution to connect to a git repository. No matter what I tried(details below) it complains about host key not present in the registry and shows me no option to accept the host key. The remote server is running Gitlab and is under my control. There's no problem with connectivity or firewalls.
What I tried so far:
connecting without giving a password, with user git
connecting while giving a password, with another user
adding manually the host key in the known_hosts file that is found in the ssh home directory(Preferences->General->Network Connections->SSH2->SSH2 home).
The message is always:
The server's host key is not cached in the registry. You
have no guarantee that the server is the computer you
think it is.
The server's rsa2 key fingerprint is:
ssh-rsa 2048 xx:xx:xx...
Connection abandoned.
RSE works without any problems, only egit gives me problems.
You could workaround the problem by not using the ssh protocol with the git server, but instead the git or http protocol.
One reason for the above message can be using a folder called "ssh" instead of ".ssh" (note the dot). Some colleague of mine experienced that, and this can easily happen when using Windows explorer, as it will silently remove the dot, when creating a folder called ".ssh". You have to use the command line instead.
I've just successfully built SparkleShare for windows according to guide:
https://github.com/wimh/SparkleShare/wiki
and exported my ssh public key to a server.
The problem is that I can't connect from a client behind a http_proxy to a public server with ssh running on a custom port. I had also problem with cloning any git server. I need to switch git:// protocol to http:// one. Any suggestion? Does anyone have similar experience?
This is a log file:
15:25:13 [SSH] ssh-agent started, PID=4380 Identity added:
C:\Users\MYUSER\AppData\Roaming\sparkleshare\sparkleshare.MYEMAIL.key
(C:\Users\sg0922706\AppData\Roaming\sparkleshare\sparkleshare.MYEMAIL.key)
15:25:34 [Fetcher][C:\Users\MYUSER\Documents\SparkleShare.tmp\share]
Fetchin g folder: ssh://MYGITUSER#MYHOST/MYPATH 15:25:34 [Fetcher]
Disabled host key checking MYHOST 15:25:34 [Cmd] git clone --progress
"ssh://MYGITUSER#MYHOST/MYPATH" "C:\Us
ers\MYUSER\Documents\SparkleShare.tmp\share" 15:25:37 [Git] Exit code
128 15:25:37 [Fetcher] Failed 15:25:37 [Fetcher] Enabled host key
checking for MYHOST
To get SparkleShare to use your proxy you will need to modify the config of the msysgit that is installed as part of SparkleShare. Navigate to C:\Program Files (x86)\SparkleShare\msysgit\etc and edit the gitconfig file in notepad and add the following line under the [http] tag
proxy = http://user:pass#proxyurl:port
modifying the url as required to match your settings. You can then use the "On my own server" option to add the http url of your repository.
I have a work around on this particular problem. I guess that you already successfully connected to your server via a simple SSH client (i.e. PuTTY)? With PuTTY you can easily configure an ssh connection via any kind of proxy (such as HTTP, SOCKS, Telnet, ..)
What you can do now is to specify a local "tunnel" (an SSH port forwarding rule) like this: L22 127.0.01:22 (see attachment). If you are using a ssh command line add the following option: -L 22:127.0.01:22.
So now as soon as your terminal is open and running you'll be able to reach your git server via the server url: ssh://git#127.0.0.1.
If your local port 22 is busy you can define the tunnet on a other port. i.e. if the 44 is not occupied: L44 127.0.0.1:22. The url to use in SparkleShare become ssh://git#127.0.0.1:44.
But it's a work around. I'm looking for a better solution.
Ok, so I have Hudson (v1.393) running in an Ubuntu VM and everything's working fine.
However I'm trying to add a Mac slave to the Ubuntu master and I've run in to a few problems.
I have set up SSH keys so that from the command line, the Ubuntu VM can ssh using the key into a user called hudson on the Mac.
In the Hudson slave configuration, I have "Launch slave agents on Unix machines via SSH" selected and have entered the host IP, username of the user on the slave and the location of my private key file on the master (which has been added to the authorised keys file on the slave).
However, the master fails to connect to the slave.
Looking at the log (below), it's trying to authenticate using a password.
Is this a fall back for a failed key based SSH attempt?
Is Hudson only trying to authenticate using a password, and I need to change something else to get it to use the key file which is defined in the configuration?
Is it just not possible to launch slave agents via ssh on a mac? (I know the name of this type of slave launch method explicity states Unix, but I was thinking (read: hoping) that it would work with OS X too)
Log
[01/14/11 10:38:07] [SSH] Opening SSH connection to 10.0.1.188:22.
[01/14/11 10:38:07] [SSH] Authenticating as hudson/******.
java.io.IOException: Password authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:319)
at com.trilead.ssh2.Connection.authenticateWithPassword(Connection.java:314)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:565)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:179)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:184)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Authentication method password not supported by the server at this stage.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:289)
... 9 more
[01/14/11 10:38:07] [SSH] Connection closed.
If anyone has managed to conquer this type of set up before, or has any tips or ideas, I'd be very grateful!
Thanks
I've recently run into the same problem, trying to launch an agent on a Mac OS X 10.6 machine using SSH.
To get password authentication to work you'll need to edit /etc/sshd_config on the client node, setting PasswordAuthentication yes
In the Hudson dashboard take the node offline, make sure the configuration has a valid username and password, and launch the agent. Also make sure that the Remote FS root directory is owned by the build user you're connecting as.
For password-less ssh authentication, first check which user the Hudson master is running as. Lets assume that this is tomcat55. Generate a public/private SSH key pair (with an empty passphrase), then verify that the Hudson user can connect.
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tomcat55/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tomcat55/.ssh/id_rsa.
Your public key has been saved in /home/tomcat55/.ssh/id_rsa.pub.
$ # authorize the hudson master on the hudson node
$ scp /home/tomcat55/.ssh/id_rsa.pub hudson#macnode:~/.ssh/authorized_keys
$ # test the connection
$ ssh -i /home/tomcat55/.ssh/id_rsa hudson#macnode
On the Hudson mac node, the /etc/sshd_config needs to allow for password-less access.
Protocol 2
PubkeyAuthentication yes
In the node configuration clear the password field, and set the private key field (in this example it is /home/tomcat55/.ssh/id_rsa). You should now be able to launch the agent:
[01/19/11 22:38:44] [SSH] Opening SSH connection to macnode:22.
[01/19/11 22:38:44] [SSH] Authenticating as hudson with /home/tomcat55/.ssh/id_rsa.
[01/19/11 22:38:45] [SSH] Authentication successful.
Check the /var/log/auth.log file on the Ubuntu machine. I'm betting you need to chmod 700 the .ssh directory of the hudson user.
I think the first answer (the selected one) is an awesome answer, but I did find a case where it is not the only solution.
In my case I have a Mac OS slave that was working and then I took that Mac down and brought up a new one. I thought I could just tweak the settings for the existing node's configuration to point it at the new Mac. It didn't work and I had all the same errors and problems described throughout this message thread.
Then I went in and deleted the node and recreated it with exactly the same settings and it worked. I suspect that SSH key fingerprint changed and by deleting the node and recreating it I was able to get it working. Whatever it is, the key component that caused it to fail is not a configuration option.