ssh keys in known hosts, but keygen -F not working - openssh

Question

I found this code elsewhere on stackoverflow:
if [ -z "`ssh-keygen -F ${wPCS_IP}`" ]; then
ssh-keyscan -p ${wPCS_PT} -H ${wPCS_IP} >> ~/.ssh/known_hosts
fi
I have two issues where I'm using the code:
This code is generating an error ($?=1) even though it succeeds.
If I run ssh-keygen -F ${wPCS_IP} again after known_hosts is appended, it does not find the keys in known_hosts, even though they were just added. This is the larger problem.
The local machine is Ubuntu Server 16.04 LTS, the remote machine is Ubuntu Server 14.04 LTS.
The major difference between my code and the code sample I found is my use of the port option -p.
Also, I've noticed that the known_hosts file does not list the machines by name or IP address. Which is different from my Gentoo laptop.

Answer 1

So it turns out that when there's an alternate port for ssh, it is stored in the known_hosts file as part of the IP address in this format:
[${WPCS_IP}]:WPCS_PT
Which means that for the if statement to work, it needs to look like this:
if ! ssh-keygen -F "[${wPCS_IP}]:${WPCS_PT}" -f ~/.ssh/known_hosts > /dev/null 2>&1; then ssh-keyscan -p ${wPCS_PT} ${wPCS-IP} >> ~/.ssh/known_hosts; fi
Thanks to alvits for getting me moving in the right direction...
Update: it turns out the Ubuntu 16.04 encrypts the IP address of the remote host (but not the port). I'm still trying to figure out how to adapt to this difference.
Another update: It turns out the the -H option is what's failing. Once you hash the key, it isn't found anymore. This works on Ubuntu 14.04:
if ! ssh-keygen -F ${IP_ADDR} -f ~/.ssh/known_hosts > /dev/null 2>&1; then ssh-keyscan -p ${PORT} ${IP_ADDR} >> ~/.ssh/known_hosts; fi
# IP_ADDR SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
# IP_ADDR SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
if ! ssh-keygen -F ${IP_ADDR} -f ~/.ssh/known_hosts > /dev/null 2>&1; then ssh-keyscan -p ${PORT} ${IP_ADDR} >> ~/.ssh/known_hosts; fi
You can see that the first if statement generates the keyscan data and the second does not because the keyscan data is correct, but if you add the -H, the keygen does not detect the hashed key entries...
However, to get a similar command to work on Ubuntu 16.04, the if has to be changed:
if ! ssh-keygen -F [${IP_ADDR}]:${PORT} -f ~/.ssh/known_hosts > /dev/null 2>&1; then ssh-keyscan -p ${PORT} ${IP_ADDR} >> ~/.ssh/known_hosts; fi
# IP_ADDR:PORT SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
# IP_ADDR:PORT SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
# IP_ADDR:PORT SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
if ! ssh-keygen -F [${IP_ADDR}]:${PORT} -f ~/.ssh/known_hosts > /dev/null 2>&1; then ssh-keyscan -p ${PORT} ${IP_ADDR} >> ~/.ssh/known_hosts; fi
In this case the known_hosts file must include the port...
-H is right out here as well. The if won't find the key if it was generated with -H.
It's frustrating that the behavior varies from version to version and that the safest hashed version doesn't work.
Yet another edit: It may be the port is specified in known_hosts when the remote server uses a non-standard port in sshd_config. This may be an expected behavior.

Answer 2

I sent an email to the open-ssh list and had several good suggestions. Basically, it is not good to rely on ssh-keygen -F and ssh-keyscan -H as file formats and locations tend to vary from system to system.
The real solution, which I will implement today is to generate certificates for each of the servers so they recognize each other. This works well for me because I have complete control over both servers.
I was given a link that explains how to setup server certificates:
https://blog.habets.se/2011/07/OpenSSH-certificates.html
Here's a link specifically for Ubuntu.
https://www.digitalocean.com/community/tutorials/how-to-create-an-ssh-ca-to-validate-hosts-and-clients-with-ubuntu
If this is my last update, assume this worked for me.