I followed the GitLab Docs to enable my project's CI to clone other private dependencies. Once it was working, I extracted from .gitlab-ci.yml:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
into a separate shell script setup.sh as follows:
which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
eval $(ssh-agent -s)
ssh-add <(echo "$SSH_PRIVATE_KEY")
mkdir -p ~/.ssh
[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
leaving only:
before_script:
- chmod 700 ./setup.sh
- ./setup.sh
I then began getting:
Cloning into '/root/Repositories/DependentProject'...
Warning: Permanently added 'gitlab.com,52.167.219.168' (ECDSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
How do I replicate the original behavior in the extracted script?
When running ssh-add either use source or . so that the script runs within the same shell, in your case it would be:
before_script:
- chmod 700 ./setup.sh
- . ./setup.sh
or
before_script:
- chmod 700 ./setup.sh
- source ./setup.sh
For a better explanation as to why this needs to run in the same shell as the rest take a look at this answer to a related question here.
Related
I have the below ci/cd script, (removed the test steps from it which works fine)
Our repo has two branches ready_4_release and master(prod), any feature branch gets merged into ready_4_release and at the end of the week we push the changes into master.
Any feature branch that gets merged into ready_4_release gets deployed into the remote airflow servers,
problem:
When a new MR gets merged into ready_4_release branch the changes are not getting deployed into remote servers but the changes on the feature branch gets merged into the ready_4_release branch, i am not able to figure out why?
When we rerun the deploy step the changes does appear on the remote server(Airflow),
OR when a new MR is submitted the previous changes gets deployed into remote servers.
Below is how the branches are setup in gitlab.
This is how the merge request settings are on the repo
stages:
- test
- deploy
.ssh-connection: &ssh-connection
- 'which git || ( apt-get update -y && apt-get install git -y )'
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- 'which rsync || ( apt-get update -y && apt-get install rsync -y )'
- eval $(ssh-agent -s)
- echo "$LOTBOT_SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh /config'
- ssh-keyscan ip-10-0-1-24.ec2.internal >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
deploy-dev:
stage: deploy
tags:
- prd-runner
image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/ubuntu
before_script:
- echo "Deploying repo dev-data-analytics to Airkube EFS"
- *ssh-connection
script:
- ssh lotbot#10.0.1.24 "sudo rm -r /mnt/airflow_dags/airkube/external_repos/dev- -data-analytics/"
- echo "${PWD}"
- cd ..
- mkdir dev-lotlinx-data-analytics && cp -R "${PWD}"/data-analytics/* "${PWD}"/dev-data-analytics
- cd dev-data-analytics
# - echo "Git commit id"
# - git show -s --format=%h
- echo "${PWD}"
- ssh lotbot#10.0.1.24 "sudo mkdir -p /mnt/airflow_dags/airkube/external_repos/dev- data-analytics"
- ssh lotbot#10.0.1.24 "sudo chown -R lotbot:sudo /mnt/airflow_dags/airkube/external_repos/dev-lotlinx-data-analytics/ && sudo chmod -R g+w /mnt/airflow_dags/airkube/external_repos/dev-data-analytics/"
- rsync -avu --delete --exclude "*__pycache__" "${PWD}"/ lotbot#10.0.1.24:/mnt/airflow_dags/airkube/external_repos/lotlinx-data-analytics;
- ssh lotbot#10.0.1.24 "sudo sleep 3s"
- ssh lotbot#10.0.1.24 "sudo chown -R lotbot:sudo /mnt/airflow_dags/airkube/external_repos/dev-lotlinx-data-analytics/ && sudo chmod -R g+w /mnt/airflow_dags/airkube/external_repos/lotlinx-data-analytics/"
rules:
- if: '$CI_COMMIT_REF_NAME == "ready_for_release" && $CI_PIPELINE_SOURCE == "push"'
Gitlab CI/CD can't connect to my remote vps.
I took https://gitlab.com/gitlab-examples/ssh-private-key as an example to make a .gitlab-ci.yaml file, its contents:
image: ubuntu
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
- eval $(ssh-agent -s)
- echo "$SSH_KEY_VU2NW" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan (domain name here) >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
Test SSH:
script:
- ssh root#(IP address here)
The runner responds with
the connection is refused
The server auth log says
sshd[2222]: Unable to negotiate with XXXXX port 53068: no matching
host key type found. Their offer: sk-ecdsa-sha2-nistp256#openssh.com
[preauth]
sshd[2220]: Unable to negotiate with XXXXX port 53068: no
matching host key type found. Their offer: sk-ssh-ed25519#openssh.com
[preauth]
Is there any way to solve this? I already tried connecting to another VPS, also without luck.
Finally got it to work, with this contents in the .gitlab-ci.yaml file:
image: ubuntu
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
- eval $(ssh-agent -s)
- mkdir -p /root/.ssh
- chmod 700 /root/.ssh
- echo "$SSH_KEY_GITLAB" >> /root/.ssh/id_rsa
- ssh-keyscan DOMAINNAME >> /root/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- chmod 400 ~/.ssh/id_rsa
Test SSH:
script:
- ssh root#DOMAINNAME
Where $SSH_KEY_GITLAB is set in Gitlabs' Settings > CICD section, and is a private key, generated by Putty, converted in Putty to an open SSH key.
The public version of this key must be in the target hosts' ~/.ssh/authorized_keys
...and DOMAINNAME must be a domain that resides on the target host, or, the DNS record should point there anyhow.
With ssh -vvv came some debugging info that pointed to the checking of ~/.ssh/id_rsa, so that's where I put the private key.
I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.
.gitlab.ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master # Run this job only on changes for stage branch
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "${shell} git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
Error: .gitlab-deploy-prod.sh: line 7: -o: command not found
How can i check if directory existing??
What i've tried.
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"
if [-d "/flask-ci-cd" ] # check if directory exists
then
eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.
Your script has some errors.
Do not use eval. No, eval does not work that way. eval is evil
When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
Take your time and research how quoting and word splitting works in shell.
git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
Read documentation for the utilities you use.
No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.
How can i check if directory existing??
Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[#]}" [ -d "/flask-ci-cd" ]; then
...
In case of directory name with spaces I would go with:
if "${shell[#]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then
Pass set -x to sh to see what's happening also on the remote side.
For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF
But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:
echo "Deploy project on server ${SERVER_URL}"
work() {
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:"${git_token}"#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token) # transfer variables you need
$(declare -f work) # transfer function you need
work # call the function.
EOF
Updated answer for future reads.
.gitlab-ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
echo "\n Cloning into remote repo..."
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git
# Create and activate virtualenv
echo "\n Creating virtual env"
python3 -m venv env
else
echo "Pulling remote repo origin..."
cd flask-ci-cd
git pull
cd ..
fi
# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate
# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF
There is a test command which is explicit about checking files and directories:
test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands
Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)
I have this .gitlab-ci.yml script --- it triggers if I push directly on the master. How can I trigger this if there's a successful merge on the master branch?
before_script:
- apt-get update -qq
- apt-get install -qq git
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_staging:
type: deploy
environment:
name: staging
url: domain.com
script:
- ssh user#domain.com "cd /server/directory && git fetch && git pull origin master && exit"
only:
- master
Like the reference notes
test:
stage: test
script: ./test
only:
- merge_requests
see https://docs.gitlab.com/ee/ci/merge_request_pipelines/index.html
I'm starting an ec2 instance from the userdata and I need to clone a repo with my ansible playbooks but it fails to clone. See details below. Can anyone help me figure this out. when I ssh to the instance after bootstrap, then clone works but not while bootstrapping.
#!/usr/bin/env bash
set -x
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cd /home/ec2-user
mkdir -p .ssh
ssh-keygen -b 2048 -t rsa -f /home/ec2-user/.ssh/codecommit -q -N ""
KEY_ID=`aws iam upload-ssh-public-key --user-name ${user_id} --ssh-public-key-body "$(cat /home/ec2-user/.ssh/codecommit.pub)" \
--query 'SSHPublicKey.SSHPublicKeyId' --output text`
echo -e "
Host git-codecommit.*.amazonaws.com
User $KEY_ID
IdentityFile /home/ec2-user/.ssh/codecommit
" >> /home/ec2-user/.ssh/config
ssh-keyscan -t rsa git-codecommit.us-east-2.amazonaws.com >> /home/ec2-user/.ssh/known_hosts
sudo chown -R ec2-user:ec2-user /home/ec2-user/.ssh
sudo chmod 700 /home/ec2-user/.ssh
sudo chmod 644 /home/ec2-user/.ssh/*
sudo chmod 600 /home/ec2-user/.ssh/codecommit*
eval "$(ssh-agent -s)"
export GIT_SSH_COMMAND="ssh -v -F /home/ec2-user/.ssh/config -o StrictHostKeyChecking=no"
export GIT_TRACE_PACKET=true
export GIT_TRACE=2
export GIT_CURL_VERBOSE=1
**sleep 60s**
git clone ssh://git-codecommit.us-east-2.amazonaws.com/v1/repos/ansible
Adding a sleep of 60 seconds before the git clone command did the trick. It seems like SSH Key uploads take a bit of time before becoming active.
sleep 60s
git clone ssh://git-codecommit.us-east-2.amazonaws.com/v1/repos/ansible
OR
for i in {1..30}; do
git clone ssh://git-codecommit.us-east-2.amazonaws.com/v1/repos/ansible
[ $? == 0 ] && break || sleep 2s; echo "keep trying ..."
done