Problems deploying code with Capistrano since upgrading to macOS 10.12 (Sierra), “Permission denied (publickey).” - macos

Question

So I just upgraded my Mac mini (Late 2012) to macOS 10.12 (Sierra) and everything seems fine, but I’m running into one odd problem deploying code with Capistrano. I get the following error:
Permission denied (publickey).
Never had this problem before in Mac OS X 10.11 (El Capitan) or any version prior to it. Why is this suddenly happening now? Full output of the failed Capistrano deployment below:
jakes_mac:SomeCode jake$ cap staging deploy
INFO [hkdgad21] Running /usr/bin/env mkdir -p /tmp/somecode/ as jake#example.com
DEBUG [hkdgad21] Command: /usr/bin/env mkdir -p /tmp/somecode/
jake#example.com's password:
INFO [hkdgad21] Finished in 5.166 seconds with exit status 0 (successful).
DEBUG Uploading /tmp/somecode/git-ssh.sh 0.0%
INFO Uploading /tmp/somecode/git-ssh.sh 100.0%
INFO [xyz20312] Running /usr/bin/env chmod +x /tmp/somecode/git-ssh.sh as jake#example.com
DEBUG [xyz20312] Command: /usr/bin/env chmod +x /tmp/somecode/git-ssh.sh
INFO [xyz20312] Finished in 0.240 seconds with exit status 0 (successful).
INFO [abcdef01] Running /usr/bin/env git ls-remote --heads git#github.com:SomeUser/SomeCode.git as jake#example.com
DEBUG [abcdef01] Command: ( GIT_ASKPASS=/bin/echo GIT_SSH=/tmp/somecode/git-ssh.sh /usr/bin/env git ls-remote --heads git#github.com:SomeUser/SomeCode.git )
DEBUG [abcdef01] Permission denied (publickey).
DEBUG [abcdef01] fatal: Could not read from remote repository.
DEBUG [abcdef01]
DEBUG [abcdef01] Please make sure you have the correct access rights
DEBUG [abcdef01] and the repository exists.
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as jake#example.com: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
SSHKit::Command::Failed: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Tasks: TOP => git:check
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as jake#example.com: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Answer 1

Seems like it’s an issue with SSH keys not being automatically added as it used to be in Mac OS X 10.11 (El Capitan). Is this expected behavior from macOS Sierra or something connected to OpenSSH?
Method 1: Add all known keys to the SSH agent.
So one solution I found is to run ssh-add with the -A option—which adds all known identities to the SSH agent using any passphrases stored in your keychain—like this:
ssh-add -A
Now this works but it won’t persist across reboots. So if you want to never worry about this again, just open up your user’s ~/.bash_profile file like this:
nano ~/.bash_profile
And add this line to the bottom:
ssh-add -A 2>/dev/null;
Now when you open a new Terminal window, all should be good!
Method 2: Add only SSH keys that are in the keychain to the agent.
So while the ssh-add -A option should work for most basic cases, I ran into an issue recently where I had 6-7 Vagrant boxes (which uses SSH keys/identities for access) setup on a machine on top of the more common id_rsa.pub in place.
Long story short, I ended up being locked out of a remote server due to too many failed tries based on SSH keys/identities since the server access was based on a password and SSH keys/identities are SSH keys/identities. So the SSH agent tried all of my SSH keys, failed and I couldn’t even get to the password prompt.
The problem is that ssh-add -A will just arbitrarily add every single SSH key/identity you have to the agent even if it’s not necessary to do so; such as in the case of Vagrant boxes.
My solution after much testing was as follows.
First, if you have more SSH keys/identities added to your agent than you need—as shown with ssh-add -l then purge them all from the agent like so:
ssh-add -D
With that done, then start the SSH agent as a background process like so:
eval "$(ssh-agent -s)"
Now, it gets weird and I am not too sure why. In some cases you can specifically add the ~/.ssh/id_rsa.pub key/identity to the agent like so:
ssh-add ~/.ssh/id_rsa.pub
Type in your passphrase, hit Return and you should be good to go.
But in other cases simply running this is enough to get the key/identity added:
ssh-add -K
If that’s all worked, type in ssh-add -l and you should see one lone SSH key/identity listed.
All good? Now open up your .bash_profile:
nano ~/.bash_profile
And add this line to the bottom; comment or remove the -A version if you have that in place:
ssh-add -K
That will allow the SSH key/identity to be reloaded to the SSH agent on each startup/reboot.
UPDATE 1: Based on davidalger’s answer I discovered a nicer, global solution that can work for all user’s on a system. Just open up the global SSH config located here via sudo:
sudo nano /etc/ssh/ssh_config
And add this line to the bottom of the file:
AddKeysToAgent yes
Did that—after removing the .bash_profile fix and all is good as well.
UPDATE 2: Apple has now added a UseKeychain option to the open SSH config options and considers ssh-add -A a solution as well.
As of macOS Sierra 10.12.2, Apple (I assume) has added a UseKeychain config option for SSH configs. Checking the man page (via man ssh_config) shows the following info:
UseKeychain
On macOS, specifies whether the system should search for
passphrases in the user's keychain when attempting to use a par-
ticular key. When the passphrase is provided by the user, this
option also specifies whether the passphrase should be stored
into the keychain once it has been verified to be correct. The
argument must be ``yes'' or ``no''. The default is ``no''.
Which boils down to Apple seeing the solution as either adding ssh-add -A to your .bash_profile as explained in this Open Radar ticket or adding UseKeychain as one of the options in a per user ~/.ssh/config.

Answer 2

Is this expected behavior from macOS Sierra or something connected to OpenSSH?
This is due to a new feature in OpenSSH 7.2 causing a change in behavior on the part of the SSH client. From the release notes:
ssh(1): Add an AddKeysToAgent client option which can be set to
'yes', 'no', 'ask', or 'confirm', and defaults to 'no'. When
enabled, a private key that is used during authentication will be
added to ssh-agent if it is running (with confirmation enabled if
set to 'confirm').
There were other interesting (security related) features introduced as well, although the release was considered mainly a bug fix release. This particular feature resulted in a default behavior change on OS X since it's default value is 'no' and OS X (I don't know about other clients) previously added keys to the agent as they were used.
So if you add the following to your ~/.ssh/config file (or the global ssh_config that should be located in /etc/ssh/ssh_config), keys will again be added to your agent as they are used.
AddKeysToAgent yes
This one-liner makes it pretty easy:
echo "AddKeysToAgent yes" >> ~/.ssh/config
After doing this, I was able to achieve the expected behavior:
$ ssh-add -l
The agent has no identities.
$ ssh -T git#bitbucket.org
logged in as davidalger.
You can use git or hg to connect to Bitbucket. Shell access is disabled.
$ ssh-add -l
2048 SHA256:<snip> (RSA)

Answer 3

This help me to resolve the issue on MacOS Serra:
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa_file