Permission denied when ansible tries to create a directory with sudo - amazon-ec2

I have a roles/ec2/tasks/main.yml that is trying to create a folder:
---
- name: Mkdir /opt/applications
file: path=/opt/applications state=directory
it is called in the roles of start.yml:
- hosts: tag_composant_XXX:&tag_Name_XXX-sandbox
remote_user: ec2-user
vars:
ec2_ami_name: XXX-base-{{ ansible_date_time.year }}-{{ ansible_date_time.month }}-{{ ansible_date_time.day }}
ec2_ami_description: Ami to launch XXX
instance_tag_environnement: XXX
roles:
- {role: ec2, sudo: true}
it is saying that
failed: [x.x.x.x] => {"failed": true, "parsed": false}
Traceback (most recent call last):
File "/home/ec2usr/.ansible/tmp/ansible-tmp-1438095761.0-196976221154211/file", line 1994, in <module>
main()
File "/home/ec2usr/.ansible/tmp/ansible-tmp-1438095761.0-196976221154211/file", line 279, in main
os.mkdir(curpath)
OSError: [Errno 13] Permission denied: '/opt/applications'
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/xxx/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 4869
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
Shared connection to x.x.x.x closed.
The execution is done via:
ansible-playbook --private-key=~/.ssh/key -vvvv -i ../ec2.py start.yml
(I have not touched the py script)
It worked before changing the ansible version (see this). What I have done more than just uninstalling + installing ansible, is that I have removed some folders in ~/.ansible/tmp/ (something like ansible-tmp-1438095761.0-196976221154211/, but I do not remember the names exactly). Is it a problem because of it?
I have managed to connect to the EC2 instance manually and create the folder, but with Ansible it seems not to work. Why? What is the problem?

Not sure if this was possible before. But one can define this directly at the task level now e.g.
- name: Mkdir /opt/applications
file:
path=/opt/applications
state=directory
become: yes
also https://docs.ansible.com/ansible/2.7/user_guide/become.html might help with further questions

Based on all the comments I am making an answer to this question:
Accordingly to the discussions on the forum of Ansible's repo there was a role level break. So it will be better to switch to 1.9.1 version. What is more, there was another change in the roles: sudo has changed to become (as mentioned in another question's answer). And that seems to fix my problem even if the docs says that sudo still works.
I have replaced:
- {role: ec2, sudo: true}
by
- {role: ec2, become: yes}

Related

connecting to Git via ssh on Windows fails

While my Linux machines can access the hosted repositories on the NAS using its hostname, my Windows machine fails to access it from command line / Visual Studio Code. During my debugging attempts I found a way so that at least TortoiseGit can access the repositories: I had to add an env var GIT_SSH_COMMAND=ssh -vvv (besides the already existing GIT_SSH=C:\Program Files\TortoiseGit\bin\TortoiseGitPlink.exe).
When I issue a git pull via cmd for a repository accessed in Git config via url = ssh://gituser#my-devices-hostname/volume1/git/reponame.git I get the following output:
OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "my-devices-hostname" port 22
debug2: ssh_connect_direct
debug1: Connecting to my-devices-hostname [192.168.0.50] port 22.
getaddrinfo: atch: Name or service not known
ssh: connect to host my-devices-hostname port 22: failure
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
When I replace the hostname by a static IP I get the same error:
OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.0.50 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.0.50 [192.168.0.50] port 22.
getaddrinfo: atch: Name or service not known
ssh: connect to host 192.168.0.50 port 22: failure
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
When doing the same with TortoiseGit, everything works fine, I get:
OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "my-devices-hostname" port 22
debug2: ssh_connect_direct
debug1: Connecting to my-devices-hostname [192.168.0.50] port 22.
debug1: Connection established.
...
I've recently set up a new DSL router (Fritz!Box 7590), maybe this is somehow related. Since I guessed that the problem could be related to IPv6, I've disabled IPv6 on the Synology NAS, but no change.
Update:
fedrik asked me to perform a manual ssh. Here are the results:
ssh gituser#my-devices-hostname
fatal: Interactive git shell is not enabled.
hint: ~/git-shell-commands should exist and have read and execute access.
Connection to my-devices-hostname closed.
and surprisingly:
ssh gituser#196.168.0.50
ssh: connect to host 196.168.0.50 port 22: Connection timed out
I updated my Git- and Tortoise installation, now I get
ssh gituser#192.168.0.50
fatal: Interactive git shell is not enabled.
hint: ~/git-shell-commands should exist and have read and execute access.
Connection to 192.168.0.50 closed.
The hostname based ssh remained the same.
Inspired by the answer from VonC, I checked if there are multiple instances of ssh installed on my system. I found the following:
C:\Windows\System32\OpenSSH\ssh.exe
C:\Program Files\Git\usr\bin\ssh.exe
So I checked (after updating Tortoise and Git for Windows), which one is used. First, I left GIT_SSH_COMMAND=ssh -vvv:
git pull
OpenSSH_8.4p1, OpenSSL 1.1.1h 22 Sep 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/c/Users/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/c/Users/user/.ssh/known_hosts2'
debug2: resolving "my-devices-hostname" port 22
debug2: ssh_connect_direct
debug1: Connecting to my-devices-hostname [192.168.0.50] port 22.
getaddrinfo: atch: Name or service not known
ssh: connect to host my-devices-hostname port 22: failure
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
"C:\Program Files\Git\usr\bin\ssh.exe" -vvv gituser#192.168.0.50
OpenSSH_8.4p1, OpenSSL 1.1.1h 22 Sep 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.0.50 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/c/Users/bjoer/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/c/Users/bjoer/.ssh/known_hosts2'
debug2: ssh_connect_direct
debug1: Connecting to 192.168.0.50 [192.168.0.50] port 22.
debug1: Connection established.
After I changed the env var: GIT_SSH_COMMAND=C:/Windows/System32/OpenSSH/ssh.exe -vvv I got this:
git pull
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug3: Failed to open file:C:/Users/user/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolving "my-devices-hostname" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to my-devices-hostname [192.168.0.50] port 22.
getaddrinfo: atch: Der angegebene Host ist unbekannt.
ssh: connect to host my-devices-hostname port 22: failure
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
C:\Windows\System32\OpenSSH\ssh.exe -vvv gituser#192.168.0.50
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug3: Failed to open file:C:/Users/user/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolve_canonicalize: hostname 192.168.0.50 is address
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.0.50 [192.168.0.50] port 22.
debug1: Connection established.
When I unset GIT_SSH and GIT_SSH_COMMAND env vars, I get this:
git pull
getaddrinfo: atch: Name or service not known
ssh: connect to host my-devices-hostname port 22: failure
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
So, I can connect via SSH to the Git server, but when issues from the Git executable itself, it fails.
When I launch up a WSL on the same machine, I can perform a git pull without problems.
Any suggestions on how to tackle this issue?
Although this issue appeared somewhat after I installed a new router, the root cause was a defective setup of the Git server on my Synology NAS. After setting up the Git server from scratch following these steps, everything worked fine again:
Setup user and folder
Install WebDAV package via Diskstation Package Manager
Configure WebDAV: Enable HTTP on port 5005 and HTTPS on port 5006
Control Panel -> File Services -> SMB -> Advanced Settings -> Set min and max SMB protocols to SMB1 and SMB3
Create user gituser via Diskstation interface (group users, no access to shared folders, with File Station and WebDAV privilages)
Add new shared folder called git (located at /volume1/git) with read/write access for gituser. This folder will hold all the repos.
Install Git Server package via Diskstation
Open Git Server and allow gituser permissions (one has to login as "admin" to see the available users)
Enable SSH access on Diskstation (Control Panel > Terminal & SNMP > Enable SSH Service)
Activate user home service (Control Panel > Users > Extended > Userbase > Activate user home service)
Configure SSH Access
create ~/.ssh folder for gituser on server
sudo su
mkdir /volume1/homes/gituser/.ssh
create a private/public key pair via PuTTYgen (SSH-2 RSA, 2048 bits)
copy the public key show at top of the window to a file named authorized_keys
copy that file from local computer to gituser account on server
scp authorized_keys username#nas:/volume1/homes/gituser/.ssh
change permissions while logged in as root
cd /volume1/homes/gituser/
sudo chown -R gituser:users .ssh
sudo chmod 700 .ssh
sudo chmod 644 .ssh/authorized_keys
sudo chmod 755 /volume1/homes/gituser
activate public key SSH access by setting the following properties in /etc/ssh/sshd_config
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys
disable and then re-enabled the SSH service (Control Panel > Terminal & SNMP > Enable SSH Service)
I was getting this error as a result of setting git config --global ssh.variant plink. By setting git config --global ssh.variant ssh, this error went way. The problem seems to be that git was passing a -batch option, which plink was interpreting as a -b option (hence the strange error involving "atch".
I would try the same ssh call without GIT_SSH set
set GIT_SSH=
With a recent Windows 10, openSSH is either directly installed or provided with Git For Windows.
Make sure to use the latest version of Git, and add its usr/bin folder to your PATH.

X11 from macOS 10.15.4 to OracleLinux7 failing

I'm trying to get x11 forwarding working from my mac to an OracleLinux7 box for Oracle setup. Here is the command and errors I'm receiving:
ssh -Y user#foo.bar.com
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
Warning: No xauth data; using fake authentication data for X11 forwarding.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_TERMINAL_VERSION = 3.3.9
debug1: Sending env LC_TERMINAL = iTerm2
debug1: Remote: X11 forwarding disabled in user configuration file.
X11 forwarding request failed on channel 0
When trying to run xeyes or xclock
[user#foo ~]$ xeyes
Error: Can't open display:
[user#foo ~]$ echo $DISPLAY
[user#foo ~]$
Below are the configs for both the mac and linux box:
OL7:
/etc/sshd/sshd_config
# Use most defaults for sshd configuration.
Subsystem sftp internal-sftp
ClientAliveInterval 180
UseDNS no
UsePAM yes
PrintLastLog no # handled by PAM
PrintMotd no # handled by PAM
TrustedUserCAKeys /etc/ssh/trustedusercakeys.pem
PasswordAuthentication no
X11Forwarding yes
X11UseLocalhost yes
ChallengeResponseAuthentication no
Mac:
Using xQuartz
~/.ssh/config
Host *
ForwardX11 yes
echo $DISPLAY
/private/tmp/com.apple.launchd.HVJpmjSBkf/org.macosforge.xquartz:0
Steps taken to fix:
xauth + on the linux machine returns
[user#foo ~]$ xauth
xauth: file /home/adahlman/.Xauthority does not exist
Using authority file /home/adahlman/.Xauthority
xauth> exit
[user#foo ~]$ xauth +
xauth: file /home/user/.Xauthority does not exist
xauth: (argv):1: unknown command "+"
I've tried changing various X11Forwarding options to yes and no but still no dice. Any ideas? Could this be a PAM issue?

Ansible fails to connect to host on some plays

I have a very simple play that I'm having a lot of issues to get the run correctly. I keep getting an SSH unreachable error when it is run. I'm getting this on 2 separate machines in different environments (although built from the same images) These are Debian 9 boxes.
Play is to simply update an internal mirror server.
- name: Update Mirror Servers
hosts: all
become: yes
gather_facts: yes
tasks:
- name: Run Update
shell: "sudo apt-mirror"
- name: Change Permissions
file:
path: /var/apt-mirror
state: directory
recurse: yes
mode: '0755'
This is being run from Ansible AWX just in case that makes any difference.
Error is as follows.
"unreachable": true,
"msg": "Failed to connect to the host via ssh: OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS 28 May 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host x.x.x.x originally x.x.x.x\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname x.x.x.x is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host x.x.x.x originally x.x.x.x\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10035\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Control master terminated unexpectedly\r\nShared connection to x.x.x.x closed.",
"changed": false
I've tried adding the ansible_ssh_user and ansible_ssh_pass as variables on the group these machines are in.
I am able to run ping successfully etc.
Identity added: /tmp/awx_187_x0lo965h/artifacts/187/ssh_key_data
(/tmp/awx_187_x0lo965h/artifacts/187/ssh_key_data)
SSH password:
BECOME password[defaults to SSH password]:
redacted | SUCCESS => {
"changed": false,
"ping": "pong"
}
I'm sure it is probably something minor that I'm missing in config somewhere. Could someone point me in the right direction?
Thanks,
Kam

SSH in git behind proxy on windows - FATAL: Connection closed by peer

By a SSH connection, I'm trying to clone a Repository from a Company BitBucket which use 7999 port (not Bitbucket.org) using Git Bash. I've generated the RSA key and added the public key into my profile of the BitBucket Company and the keys re located in ~/.ssh, I've setup the proxy by using git config --global http.proxy http://userPrx:pwdPrx#ipProx:8080 (because I'm under the Company Proxy) and also I have setup my config file as this post suggest. Then, when I try to Test the connection I get this:
$ ssh -vT globaldevtools -p 7999
OpenSSH_7.3p1, OpenSSL 1.0.2j 26 Sep 2016
debug1: Reading configuration data /c/Users/MyUser/.ssh/config
debug1: /c/Users/MyUser/.ssh/config line 5: Applying options for globaldevtools
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: Executing proxy command: exec /C/Users/MyUser/AppData/Local/Programs/Git/mingw64/bin/connect.exe -S IpProxy:8080 x.x.x.x 7999
debug1: permanently_drop_suid: 1104711
debug1: identity file /c/Users/MyUser/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/MyUser/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
FATAL: Connection closed by peer.
ssh_exchange_identification: Connection closed by remote host
This is my config file:
ProxyCommand /C/Users/MyUser/AppData/Local/Programs/Git/mingw64/bin/connect.exe -S IpProxy:8080 %h %p
Host globaldevtools
User git
Port 7999
Hostname x.x.x.x
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
I must indicate that in this file (config) instead of IpProxy:8080 I've tried with
http://IpProxy:8080
http://usrProx:pwdProx#IpProxy:8080
usrProx:pwdProx#IpProxy:8080
Do I have to do something else? Did I miss something? All help is appreciated.

In Gitlab CI build, I can't ssh into AWS EC2 by private key

At first, I was trying to use ansible for deployment after gitlab CI built, but it shows "host unreachable" in result.
After some trial and error, I found out the problem is ssh permission denied when ssh by private key into my AWS EC2 instance to deploy.
My .gitlab-ci.yml config is something like this:
.gitlab-ci.yml
image: ansible/ubuntu14.04-ansible:stable
stages:
- deploy
deploy_web:
stage: deploy
script:
- "echo Ansible"
- "echo Environment: ${ENV}"
- "echo TAG: ${TAG}"
- "echo ${VAULT_PASS} > vault_pass.txt"
- "mkdir sshkey"
- "echo ${SSH_KEY_APP} > ./sshkey/app-key.pem"
- "chmod 600 ./sshkey/app-key.pem"
- "export SSH_KEY_DIR=`pwd`/sshkey"
- "export ANSIBLE_HOST_KEY_CHECKING=False"
- "ssh-keyscan foobar.io >> ~/.ssh/known_hosts"
- "ssh -v -i ./sshkey/app-key.pem ubuntu#foobar.io" // for debugging
- "ansible-playbook -i ${ENV} servers.yml --vault-password-file vault_pass.txt -vvvv --tags=${TAG}"
When gitlab CI builds this, it basically gives these ssh error messages:
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to foobar.io [12.34.56.78] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file ./sshkey/app-key.pem type -1
debug1: identity file ./sshkey/app-key.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA be:b1:53:76:aa:bf:65:ea:b4:1b:7a:8f:cc:7c:2a:79
debug1: Host 'foobar.io' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
Warning: Permanently added the ECDSA host key for IP address '12.34.56.78' to the list of known hosts.
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: ./sshkey/app-key.pem
debug1: key_parse_private2: missing begin marker
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: No more authentication methods to try.
Permission denied (publickey).
Also tried using absolute path:
$ cat /builds/foobar/bar/sshkey/app-key.pem
-----BEGIN RSA PRIVATE KEY-----
...(the key)...
-----END RSA PRIVATE KEY-----
$ ssh -v -i /builds/foobar/bar/sshkey/app-key.pem ubuntu#foobar.io
Permission denied (publickey).
These are what I have tried:
try using shell executor for gitlab CI runner -> failed
run the scripts in a local docker container -> success
ssh into the runner instance manually (not through CI) and run the scripts in shell -> success
ssh into the runner instance manually and run the scripts in docker container -> success
As a conclusion - It only fails when run by gitlab CI, so I wonder if there are any additional configuration I haven't noticed to do things like this...
Many thanks for anyone can help!
The real problem is
When echo-ing a multiline environment variable, quotes are needed.
So basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh.
If it fails when run by GilabCI, it means the user used by GitLab CI is not the same as the one used when you ssh in the running instance.
See for instance "AWS SSH connection error: Permission denied (publickey)"
Another thing to check is PermitRootLogin and AllowUsers in /etc/ssh/sshd_config.
This debug1: key_parse_private2: missing begin marker appears even after successful key authorization if your user access restricted.
Check after a manually ssh on the remote machine:
tail -f -n 80 /var/log/auth.log
The OP DarkBtf adds in the comments:
When echo-ing a multiline environment variable, quotes are needed.
So basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh.

Resources