Gitlab CI check if directory exists before pulling origin - bash

Question

I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.
.gitlab.ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master # Run this job only on changes for stage branch
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "${shell} git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
Error: .gitlab-deploy-prod.sh: line 7: -o: command not found
How can i check if directory existing??
What i've tried.
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"
if [-d "/flask-ci-cd" ] # check if directory exists
then
eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.

Answer 1

Your script has some errors.
Do not use eval. No, eval does not work that way. eval is evil
When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
Take your time and research how quoting and word splitting works in shell.
git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
Read documentation for the utilities you use.
No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.
How can i check if directory existing??
Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[#]}" [ -d "/flask-ci-cd" ]; then
...
In case of directory name with spaces I would go with:
if "${shell[#]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then
Pass set -x to sh to see what's happening also on the remote side.
For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF
But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:
echo "Deploy project on server ${SERVER_URL}"
work() {
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:"${git_token}"#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token) # transfer variables you need
$(declare -f work) # transfer function you need
work # call the function.
EOF

Answer 2

Updated answer for future reads.
.gitlab-ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
echo "\n Cloning into remote repo..."
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git
# Create and activate virtualenv
echo "\n Creating virtual env"
python3 -m venv env
else
echo "Pulling remote repo origin..."
cd flask-ci-cd
git pull
cd ..
fi
# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate
# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF

Answer 3

There is a test command which is explicit about checking files and directories:
test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands
Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)