Odd ansible behaviour in CentOS container - ansible

Question

I have some odd behaviour when using ansible inside a CentOS 8 base container. All I am doing initially is testing basic function, essentially run a ping from another machine using ansible from a gitlab runner. It should be super simple, but I'm having issues with basic auth.
I've set up authorized keys and checked to make sure they work for the connection from the container host (Centos8 with podman) to the test machine also CentOS8, all working correctly with ansible see below:
[root#automation home]# ansible all -i lshyp01.lab, -u ansible -v --private-key=/home/ansible/.ssh/id_rsa -a "/usr/sbin/ping -c 3 8.8.8.8"
Using /etc/ansible/ansible.cfg as config file
lshyp01.lab | CHANGED | rc=0 >>
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=5.30 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=5.21 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=4.97 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.967/5.160/5.304/0.153 ms
[root#automation home]#
however when I run the same command via the Gitlab runner I get:
$ useradd ansible
$ mkdir -p /home/ansible/.ssh
$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' > /home/ansible/.ssh/id_rsa
$ chmod -R 744 /home/ansible/.ssh/id_rsa*
$ chown ansible:ansible -R /home/ansible/.ssh
$ export ANSIBLE_HOST_KEY_CHECKING=False
$ ansible all -i lshyp01.lab, -u ansible -v --private-key=/home/ansible/.ssh/id_rsa -a "/usr/sbin/ping -c 3 8.8.8.8"
Using /etc/ansible/ansible.cfg as config file
lshyp01.lab | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Warning: Permanently added 'lshyp01.lab,10.16.4.19' (ECDSA) to the list of known hosts.\r\nansible#lshyp01.lab: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).",
"unreachable": true
}
Cleaning up file based variables
00:00
ERROR: Job failed: exit status 1
And here is the .gitlab-ci.yml file:
# Use minimal CentOS7 image
image: centos:latest
# Set up variables
# TF_ROOT: ${CI_PROJECT_DIR}/
# TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/state/prod
stages:
- prepare
- validate
- build
- deploy
before_script:
# Install tools - these should be baked into the image for prod
- which ssh-agent || (dnf -y install openssh-clients)
- eval $(ssh-agent -s)
- dnf -y install which
- which git || (dnf -y install git)
- which terraform || (dnf install -y dnf-utils && dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo && dnf -y install terraform)
- which ansible || (dnf -y install epel-release && dnf -y install ansible)
- which nslookup || (dnf -y install bind-utils)
- which sudo || (dnf -y install sudo)
# Seup user
- useradd ansible
- mkdir -p /home/ansible/.ssh
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' > /home/ansible/.ssh/id_rsa
- chmod -R 744 /home/ansible/.ssh/id_rsa*
- chown ansible:ansible -R /home/ansible/.ssh
# Pre testing
sshtest:
stage: prepare
script:
- export ANSIBLE_HOST_KEY_CHECKING=False
- ansible all -i lshyp01.lab, -u ansible -v --private-key=/home/ansible/.ssh/id_rsa -a "/usr/sbin/ping -c 3 8.8.8.8"
I have verified that the key is correct. Any help is greatly appreciated.

Answer 1

The answer turned out to be an issue with Gitlab variables. In the end I had to encode the keys into base 64 to store them then decode them on use. the updated gitlab-ci section is below.
As pointed out the above example also had the wrong permissions, however, I'd tried a few options, I should have reverted the permission changes before posting, sorry for the confusion.
- mkdir -p /root/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > /root/.ssh/id_rsa
- echo "$SSH_PUBLIC_KEY" | base64 -d > /root/.ssh/id_rsa.pub
- chmod -R 600 /root/.ssh/id_rsa && chmod -R 664 /root/.ssh/id_rsa.pub
- export ANSIBLE_HOST_KEY_CHECKING=False