I am writing a bitbucket pipeline to deploy my angular project to the ec2 instance. This is my pipeline using rsync.
image: node:12.18.3
pipelines:
branches:
dev:
- step:
name: Build Test Environment
caches:
- node
script:
- npm install
- npm run build-qa
artifacts:
- dist/qa/**
deployment: test
- step:
name: Deploy
trigger: manual
script:
- apt-get update && apt-get install -y rsync
- ssh-keyscan -H $SERVER >> ~/.ssh/known_hosts
- cd $BITBUCKET_CLONE_DIR/dist/qa
- ls
- rsync -v -e ssh . $SSH_USER#$SERVER:/var/www/html/myproject
- echo "Deployment is done...!"
But this is giving me this error.
+ rsync -v -e ssh . $SSH_USER#$SERVER:/var/www/html/myproject
skipping directory .
rsync: link_stat "/opt/atlassian/pipelines/agent/build/dist/qa/$SSH_USER#myip" failed: No such file or directory (2)
rsync: link_stat "/opt/atlassian/pipelines/agent/build/dist/qa/ecdsa-sha2-nistp256" failed: No such file or directory (2)
rsync: change_dir#3 "/opt/atlassian/pipelines/agent/build/dist/qa//AAAAE2VjZHNhLXNtYTItbmlzdHAyNTwAAqAIbmlzdHAyNsYAAABBBGqKvzLI7IolhgM1ZEfol3VuJX4CX6jzqSyM6AzUgPbpyERywu/7U/SioMc/SLeJyfhYnWAJVApt8oOsqIjLqDg=:/var/www/html/myproject" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(713) [Receiver=3.1.2]
I tried a lot to find out a solution to this I even tried with the rsync-deploy pipe but it also gives the same above error can someone help me to write this pipeline correctly to get my requirement done?
Seems similar to the issue here https://serverfault.com/questions/363555/why-is-rsync-skipping-the-main-directory
you probably have folders inside you want to recursively include in the transfer
you need to change the source to ./ so it gets it is a folder
and as a bonus
you can/should just run ls with the path, and same with rsync, and not cd into there. Remember the trailing /
Related
I have defined the following stages, environment variable in my .gitlab-ci.yaml script:
stages:
- prepare
- run-test
variables:
MY_TEST_DIR: "$HOME/mytests"
prepare-scripts:
stage: prepare
before_script:
- cd $HOME
- pwd
script:
- echo "Your test directory is $MY_TEST_DIR"
- cd $MY_TEST_DIR
- pwd
when: always
tags:
- ubuntutest
When I run the above, I get the following error even though /home/gitlab-runner/mytests exists:
Running with gitlab-runner 15.2.1 (32fc1585)
on Ubuntu20 sY8v5evy
Resolving secrets
Preparing the "shell" executor
Using Shell executor...
Preparing environment
Running on PCUbuntu...
Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /home/gitlab-runner/tests/sY8v5evy/0/childless/tests/.git/
Checking out cbc73566 as test.1...
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ cd $HOME
/home/gitlab-runner
$ echo "Your test directory is $MY_TEST_DIR"
SDK directory is /mytests
$ cd $MY_TEST_DIR
Cleaning up project directory and file based variables
ERROR: Job failed: exit status 1
Is there something that I'm doing wrong here? Why is $HOME empty/NULL when used along with other variable?
When setting a variable using gitlab-ci variables: directive, $HOME isn't available yet because it's not running in a shell.
$HOME is set by your shell when you start the script (or before_script) part.
If you export it during the script step, it should be available, so :
prepare-scripts:
stage: prepare
before_script:
- cd $HOME
- pwd
script:
- export MY_TEST_DIR="$HOME/mytests"
- echo "Your test directory is $MY_TEST_DIR"
- cd $MY_TEST_DIR
- pwd
when: always
tags:
- ubuntutest
I am trying to create a CI/CD pipeline to build war file and deploy it to EC2 from GitLab.
Once the war file is created, I would like to copy it to some folder in EC2 so that from there I would like to copy it to tomcat server.
The following is the ".gitlab-ci.yml" file.
stages:
- build
- deploy
build:
stage: build
image: maven:3-jdk-8
script:
- mvn install
artifacts:
paths:
- target/
deploy:
stage: deploy
before_script:
# Generate SSH Key
- mkdir -p ~/.ssh
- echo -e "$EC2_SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- scp target/gitlabec2pipeline.war ec2-user#$EC2_DEPLOY_SERVER:/gitlabec2pipeline.war
- bash .gitlab-deploy-ec2.sh
I have added the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY variables.
But when the above pipeline is run, in the deploy stage the scp command is giving "permission denied" error.
Any idea on how to solve this?
Error Message:
Running with gitlab-runner 14.5.2 (e91107dd)
on blue-3.shared.runners-manager.gitlab.com/default zxwgkjAP
Resolving secrets
00:00
Preparing the "docker+machine" executor
Using Docker executor with image ruby:2.5 ...
Pulling docker image ruby:2.5 ...
Using docker image sha256:27d049ce98db4e55ddfaec6cd98c7c9cfd195bc7e994493776959db33522383b for ruby:2.5 with digest ruby#sha256:ecc3e4f5da13d881a415c9692bb52d2b85b090f38f4ad99ae94f932b3598444b ...
Preparing environment
00:01
Running on runner-zxwgkjap-project-31676452-concurrent-0 via runner-zxwgkjap-shared-1639429231-955193ca...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/te2122/deploytoaws/.git/
Created fresh repository.
Checking out dc27fd6f as master...
Skipping Git submodules setup
Downloading artifacts
00:02
Downloading artifacts for build (1880203000)...
Downloading artifacts from coordinator... ok id=1880203000 responseStatus=200 OK token=9RSALYus
Executing "step_script" stage of the job script
00:02
Using docker image sha256:27d049ce98db4e55ddfaec6cd98c7c9cfd195bc7e994493776959db33522383b for ruby:2.5 with digest ruby#sha256:ecc3e4f5da13d881a415c9692bb52d2b85b090f38f4ad99ae94f932b3598444b ...
$ mkdir -p ~/.ssh
$ echo -e "$EC2_SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
$ chmod 600 ~/.ssh/id_rsa
$ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ scp target/gitlabec2pipeline.war ec2-user#$EC2_DEPLOY_SERVER:/gitlabec2pipeline.war
Warning: Permanently added '54.205.169.131' (ECDSA) to the list of known hosts.
scp: /gitlabec2pipeline.war: Permission denied
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
Thank you.
I have added the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY variables.
AWS keys don't matter here since you're using SSH key for authentication when you use ssh (or scp) to connect to the EC2 instance.
If you are sure you're using the correct key value, your environment variable $EC2_SSH_PRIVATE_KEY is not well-formed. Common issues come up with newlines in the variable and use of CRLF (often can be added when copy/pasting the key in the web UI) instead of LF.
To work around this more reliably, you could:
use a file type variable, then copy the file into ~/.ssh OR
base64 encode your key before putting it into the CI variable. This avoids any possible issues with newlines and CRLF. Then decode it in the job, placing the value into the file.
Update:
It looks like your user does not have permission to the destination directory on the server. You need to either give appropriate permission to the ec2-user or choose a different destination location where the user does have permission to write files.
I am attempting to use a shared runner to run a script which handles env vars necessary for deployment. The section of my YAML config that is failing is:
release:
stage: release
image: docker:latest
only:
- master
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
before_script:
- docker version
- docker info
- docker login -u ${CI_REGISTRY_USER} -p ${CI_BUILD_TOKEN} ${CI_REGISTRY}
script:
- dckstart=$(cat dockerfile-start)
- export > custom_vars
- chmod +x scripts/format-variables.sh
- bash scripts/format-variables.sh
- dckenv=$(cat custom_vars)
- dckfin=$(cat dockerfile-finish)
- echo -e "$dckstart\n$dckenv\n$dckfin" >> Dockerfile
- rm dockerfile-start dockerfile-finish custom_vars
- docker build -t ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest --pull .
- docker push ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest
after_script:
- docker logout ${CI_REGISTRY}
This step fails & gives the error:
$ chmod +x scripts/format-variables.sh
$ bash scripts/format-variables.sh
/bin/sh: eval: line 101: bash: not found
I have attempted:
/bin/bash scripts/format-variables.sh
/bin/sh: eval: line 114: /bin/bash: not found
cd scripts && ./format-variables.sh
/bin/sh: eval: line 116: ./format-variables.sh: not found
--shell /bin/bash scripts/format-variables.sh
/bin/sh: eval: line 114: --shell: not found
The final attempt was an idea I grabbed from the docs. I have not specified the shared runners to use but I assume the one being used is UNIX based as all other UNIX commands work.
Is it possible to do this via a shared runner or do I need to get a dedicated runner for this?
NOTE: I have to use Bash for this script & not Shell due to using arrays. If I were to use Shell, I would come up with the error mentioned here
The docker:latest image doesn't contain bash, to save space. You can either install it (see How to use bash with an Alpine based docker image?) or use a different base image (like CentOS or Ubuntu).
Use an image that has bash installed like CentOS or Ubuntu.
My problem is the bash script I created got this error "/bin/sh: eval: line 88: ./deploy.sh: not found" on gitlab. Below is my sample script .gitlab-ci.yml.
I suspect that gitlab ci is not supporting bash script.
image: docker:latest
variables:
IMAGE_NAME: registry.gitlab.com/$PROJECT_OWNER/$PROJECT_NAME
DOCKER_DRIVER: overlay
services:
- docker:dind
stages:
- deploy
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- docker pull $IMAGE_NAME:$CI_BUILD_REF_NAME || true
production-deploy:
stage: deploy
only:
- master#$PROJECT_OWNER/$PROJECT_NAME
script:
- echo "$PRODUCTION_DOCKER_FILE" > Dockerfile
- docker build --cache-from $IMAGE_NAME:$CI_BUILD_REF_NAME -t $IMAGE_NAME:$CI_BUILD_REF_NAME .
- docker push $IMAGE_NAME:$CI_BUILD_REF_NAME
- echo "$PEM_FILE" > deploy.pem
- echo "$PRODUCTION_DEPLOY" > deploy.sh
- chmod 600 deploy.pem
- chmod 700 deploy.sh
- ./deploy.sh
environment:
name: production
url: https://www.example.com
And this also my deploy.sh.
#!/bin/bash
ssh -o StrictHostKeyChecking=no -i deploy.pem ec2-user#targetIPAddress << 'ENDSSH'
// command goes here
ENDSSH
All I want is to execute deploy.sh after docker push but unfortunately got this error about /bin/bash thingy.
I really need your help guys. I will be thankful if you can solve my problem about gitlab ci bash script got error "/bin/sh: eval: line 88: ./deploy.sh: not found".
This is probably related to the fact you are using Docker-in-Docker (docker:dind). Your deploy.sh is requesting /bin/bash as the script executor which is NOT present in that image.
You can test this locally on your computer with Docker:
docker run --rm -it docker:dind bash
It will report an error. So rewrite the first line of deploy.sh to
#!/bin/sh
After fixing that you will run into the problem that the previous answer is addressing: ssh is not installed either. You will need to fix that too!
docker:latest is based on alpine linux which is very minimalistic and does not have a lot installed by default. For example, ssh is not available out of the box, so if you want to use ssh commands you need to install it first. In your before_script, add:
- apk update && apk add openssh
Thanks. This worked for me by adding bash
before_script:
- apk update && apk add bash
Let me know if that still doesn't work for you.
I'm using LFTP on Gitlab CI to deploy a set of files. I've got this working nicely on one server that I've set up (a staging server using SFTP). However, on my client's server, I can't seem to connect. The server is setup using FTP and I have to use plain/unsecure mode to connect via Filezilla - it does connect and work fine (although I'll be giving some advice to use SFTP in the future).
When I try to do the same using LFTP through the .gitlab-ci.yml file I get the following error:
Unknown command `ftp.example.com'.
mirror: Not connected
ERROR: Build failed: exit code 1
I suspect that this is because of using plain FTP but I've tried changing hosts, putting ftp:// infront of the host and a few other commands using set but having no luck.
Here's (an edited version of) my .gitlab-ci.yml file:
stages:
- build-staging
- build-production
variables:
EXCLUDE: "--exclude '.htaccess' --exclude-glob .git* --exclude '.git/' --exclude 'wp-config.php'"
SOURCE_DIR: "./"
# STAGING
DEST_DIR: "/"
HOST_STAGING: "sftp://123.456.789"
USERNAME_STAGING: "user"
PASSWORD_STAGING: "password"
# PRODUCTION
DEST_DIR_PROD: "/"
HOST_PROD: "ftp.example.com"
USERNAME_PROD: "user"
PASSWORD_PROD: "password"
job1:
stage: build-staging
environment: staging
script:
- apt-get update -qq && apt-get install -y -qq lftp
- echo "Deploying"
- lftp -c "set ftp:ssl-allow no; set sftp:auto-confirm yes; open -u $USERNAME_STAGING,$PASSWORD_STAGING $HOST_STAGING; mirror -Rv --ignore-time --parallel=10 $EXCLUDE $SOURCE_DIR $DEST_DIR_STAGING"
only:
- staging
tags:
- 2gb
job2:
stage: build-production
environment: production
when: manual
script:
- apt-get update -qq && apt-get install -y -qq lftp
- echo "Deploying"
- lftp -c "set ftp:ssl-allow no; open -u $USERNAME_PROD,$PASSWORD_PROD $HOST_PROD; mirror -Rv --ignore-time --parallel=10 $EXCLUDE $SOURCE_DIR $DEST_DIR_PROD"
only:
- production
tags:
- 2gb
Any help would be great, thanks!
This was due to a special characters in the password - my password ended with & which caused lftp to expect a different command. To fix this, I removed the quotes and escaped the & with a |, like so:
PASSWORD_PROD: password\&