I want to write a .bowerrc file at the start of my pipeline.
echo '{ "allow_root": true }' > /root/.bowerrc
How do I escape all the quotes and colons to make it valid in yaml?
image: java:8
pipelines:
default:
- step:
script:
- echo '{ "allow_root": true }' > /root/.bowerrc
just remove the whitespaces
- npm install -g bower && echo '{"allow_root":true}' > /root/.bowerrc && npm install -g grunt-cli
Related
I have built a Golang project, I want to deploy the app when successfully tested with GitLab ci, but when do some tests, it fails because cannot connect to MySQL.
I want to use the Golang image and MySQL image in one stage.
This is my current pipeline. On stage test, on before script fails (/bin/bash: line 130: mysql: command not found)
# To contribute improvements to CI/CD templates, please follow the Development guide
at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml
image: golang:latest
services:
- mysql:latest
stages:
- test
- build
- deploy
variables:
MYSQL_DATABASE: "db"
MYSQL_USER: "user"
MYSQL_PASSWORD: "password"
MYSQL_ROOT_PASSWORD: "password"
format:
stage: test
variables:
# Configure mysql environment variables (https://hub.docker.com/_/mysql/)
MYSQL_DATABASE: $MYSQL_DATABASE
MYSQL_PASSWORD: $MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
services:
- mysql:latest
before_script:
- mysql --version
script:
- go fmt $(go list ./... | grep -v /vendor/)
- go vet $(go list ./... | grep -v /vendor/)
- go test -race $(go list ./... | grep -v /vendor/)
compile:
stage: build
script:
- mkdir -p typing
- go build -o typing ./...
artifacts:
paths:
- typing
deploy:
image: google/cloud-sdk:alpine
stage: deploy
allow_failure: true
script:
- "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- (echo "$SSH_PRIVATE_KEY" | base64 -d) > file
- echo "$SSH_PUBLIC_KEY" > file.pub
- chmod 600 file
- echo $SERVICE_ACCOUNT > file.json
- gcloud auth activate-service-account --key-file file.json
- gcloud compute scp typing/* --project="project-id" --zone="zone" vm-name:/home/ubuntu
- ssh -i file ubuntu#public-ip 'sudo ./kill.sh; sudo ./start.sh'
artifacts:
paths:
- typing
How can I achieve that?
Thanks in advance.
In the stage test, the job is based on the golang image as a result it does not come packaged with the MySQL client.
In order to reach the MySQL service you defined you need to install the client
If I am not mistaken the Golang image is based on debian, so something like this
before_script:
- apt get-update
- apt get-install -y default-mysql-client
- mysql --version
Inside .gitlab-ci.yml we define a variable (which is just the artifactId name for the project) ARTIFACT_ID: myMicroservice-1
This variable ARTIFACT_ID is sent to a general microservice which has all the scripts to publish/deploy docker, etc.
How can I read this variable direct from POM file?
pom:
<artifactId>myMicroservice-1</artifactId>
.gitlab-ci.yml:
variables:
SKIP_UNIT_TESTS_FLAG: "true"
ARTIFACT_ID: myMicroserverName
IS_OSL: "true"
KUBERNETES_NAMESPACE: test
Here is how we do it.
Value is extracted from the pom.xml based on its XPath.
We use xmllint tool from libxml2-utils, but there are various other tools for that.
Then value is saved as an environment variable in a file, which is passed to further GitLab jobs as artifact.
stages:
- prepare
- build
variables:
VARIABLES_FILE: ./variables.txt # "." is required for sh based images
POM_FILE: pom.xml
get-version:
stage: prepare
image: ubuntu
script:
- apt-get update
- apt-get install -y libxml2-utils
- APP_VERSION=`xmllint --xpath '/*[local-name()="project"]/*[local-name()="version"]/text()' $POM_FILE`
- echo "export APP_VERSION=$APP_VERSION" > $VARIABLES_FILE
artifacts:
paths:
- $VARIABLES_FILE
build:
stage: build
image: docker:latest
script:
- source $VARIABLES_FILE
- echo "Here use $APP_VERSION as you like"
This is the script to grab information from pom, in this case, artifactId:
- export myARTIFACT_ID=$(mvn exec:exec -q -Dexec.executable=echo -Dexec.args='${project.artifactId}')
- if [[ "$myARTIFACT_ID" == *finishWithWhatEverName]]; then export myVariable="false"; else export myVariable="true";
Then you can use myVariable to whatever you want.
I tried Angel's solution, but got an error:
/bin/bash: line 87: mvn: command not found
Finally I succeeded when I used the following to extract the tag value:
- export ARTIFACT_ID=$(cat pom.xml | grep "<artifactId>" | head -1 | cut -d">" -f2 | cut -d"<" -f1 | awk '{$1=$1;print}')
jobname:
stage: stage
before_script:
- export "MAVEN_ID=$(mvn help:evaluate -Dexpression=project.id -q -DforceStdout)"
- >
IFS=: read -r MAVEN_GROUPID MAVEN_ARTIFACTID MAVEN_PACKAGING MAVEN_VERSION <<< ${MAVEN_ID}
script:
- >
echo -e "groupId: ${MAVEN_GROUPID}\nartifactId: ${MAVEN_ARTIFACTID}\nversion: ${MAVEN_VERSION}\npackaging: ${MAVEN_PACKAGING}"
mvn help:evaluate -Dexpression=project.id -q -DforceStdout prints artifact identification information in the following format: com.group.id:artifactid:packaging:version
MAVEN_ID variable is parsed using IFS based on the colon (:) as a separator to get common maven variables like artifactId, groupId, version and packaging (explanation)
later these variables can be used in the code, for example for echoing values
IFS is a bash feature, thus corresponding GitLab runner should have bash installed
For some reason when I use an if statement it will not execute the configure for it. Here is my full travis.yml below.
travis.yml:
language: php
php:
- '5.6.32'
- '7.0.26'
- '7.1.12'
- '7.2.0'
os:
- windows
- linux
git:
depth: 1
matrix:
fast_finish: true
sudo: false
before_install:
- if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2.0" ]]; then git clone -b stable https://github.com/jedisct1/libsodium.git; fi
- if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2.0" ]]; then cd libsodium && sudo ./configure && sudo make check && sudo make install && cd ..; fi
install:
- if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2.0" ]]; then pecl install libsodium; fi
- if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2.0" ]]; then echo "extension=sodium.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini; fi
- travis_retry composer install --no-interaction
- wget -c -nc --retry-connrefused --tries=0 https://github.com/satooshi/php-coveralls/releases/download/v1.0.1/coveralls.phar
- chmod +x coveralls.phar
- php coveralls.phar --version
before_script:
- mkdir -p build/logs
- ls -al
script:
- ./vendor/bin/phpunit --coverage-clover build/logs/clover.xml
after_success:
- travis_retry php coveralls.phar -v
branches:
only: master
cache:
directories:
- vendor
- $HOME/.cache/composer
So for some reason
- if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2.0" ]]; then
export git clone -b stable https://github.com/jedisct1/libsodium.git;
fi
does not work and I got this solution from this
My goal is to execute certain lines on certain PHP version.
Is there anything I missed?
EDIT:
you extracted the first 3 characters of the TRAVIS_PHP_VERSIONand compared it with 5 characters.. of course that doesn't work. you can either try:
if [[ ${TRAVIS_PHP_VERSION:0:5} == "7.2.0" ]]
or
if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2" ]]
END EDIT
I had some problems running scripts inside the Travis config file, since i wanted blank lines and comments and Travis got confused with that. So in general i would recommend to do the scripting in a separate bash file.
The easiest way is to do all the bash scripting in a script file and use the script files in your travis.yml
before_install: ./travis-scripts/before_install.sh
Now you can write your scripts with the bash syntax and they work right away.
If you still want to write the scripts inside the travis file, try that (not everything worked for me everytime):
install: >
if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2" ]]; then pecl install libsodium; fi;
if [[ ${TRAVIS_PHP_VERSION:0:3} == "7.2" ]]; then echo "extension=sodium.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini; fi;
travis_retry composer install --no-interaction;
wget -c -nc --retry-connrefused --tries=0 https://github.com/satooshi/php-coveralls/releases/download/v1.0.1/coveralls.phar;
chmod +x coveralls.phar;
php coveralls.phar --version;
Or leave it as it is and only put the if statements in quotes.
I'm trying to do a condition in my tests that print echo "showing dev branch" if my branch name is development but I'm receiving this error
if [ "${CIRCLE_BRANCH}" == "development"]; then echo "showing dev branch" fi
bash: -c: line 2: syntax error: unexpected end of file
if [ "${CIRCLE_BRANCH}" == "development"]; then echo "showing dev branch" fi returned exit code 1
See my circle.yml below:
general:
artifacts:
- "test_evidences"
branches:
only:
- development
machine:
node:
version: 6.10.3
dependencies:
pre:
- curl -L -o google-chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- sudo dpkg -i google-chrome.deb
- sudo sed -i 's|HERE/chrome\"|HERE/chrome\" --disable-setuid-sandbox|g' /opt/google/chrome/google-chrome
- rm google-chrome.deb
- npm install
- npm install -g grunt grunt-cli
override:
- node_modules/.bin/webdriver-manager update
test:
pre:
- sleep 60
override:
- if [ "${CIRCLE_BRANCH}" == "development"]; then
echo "showing dev branch"
fi
- grunt apiTests
- node_modules/.bin/protractor conf.js
- sed -i -- 's,//,/,g' test_evidences/htmlReport.html
Problem solved!
My new circle.yml file is:
#!/usr/bin/env bash
general:
artifacts:
- "test_evidences"
branches:
only:
- development
machine:
node:
version: 6.10.3
dependencies:
pre:
- curl -L -o google-chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- sudo dpkg -i google-chrome.deb
- sudo sed -i 's|HERE/chrome\"|HERE/chrome\" --disable-setuid-sandbox|g' /opt/google/chrome/google-chrome
- rm google-chrome.deb
- npm install
- npm install -g grunt grunt-cli
override:
- node_modules/.bin/webdriver-manager update
test:
pre:
- sleep 60
override:
- if [ "${CIRCLE_BRANCH}" == "development" ]; then
echo "showing dev branch";
fi
- grunt apiTests
- node_modules/.bin/protractor conf.js
- sed -i -- 's,//,/,g' test_evidences/htmlReport.html
I have a deploy script I only want to run if my test is successful but believe there is an issue with my conditional statement if [ "$VALID" ]
#!/bin/bash
# install dependencies
echo 'INSTALLING YARN'
npm install yarn -g
echo "INSTALLING DEPENDENCIES"
yarn install
echo "TESTING"
VALID="$(npm test)"
if [ "$VALID" ]
then
# ZIP up the code
echo 'INSTALLING ZIP'
apt-get update
echo "y" | apt-get install zip
echo 'ZIPPING'
zip -r ./Lambda-Image-Compression.zip index.js node_modules
# install aws cli so we can deploy code
echo 'INSTALLING PIP'
# echo "y" | apt-get install python-pip
echo "y" | apt-get install python-pip python-dev build-essential
echo "y" | pip install --upgrade pip
# echo "y" | sudo pip install --upgrade virtualenv
echo 'INSTALLING AWSCLI'
pip install awscli
# Copy config file to root so AWS config & credentials are set
echo 'MAKING AWS CREDENTIALS'
CREDENTIALS="[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY"
touch .aws/credentials
"$CREDENTIALS" > .aws/credentials
echo "MOVING AWS CONFIG"
cp -R .aws ~/.
# Upload to AWS
echo 'UPDATING LAMBDA FUNCTION'
aws lambda update-function-code \
--function-name resizeHandler \
--zip-file fileb://Lambda-Image-Compression.zip \
--region ap-southeast-2
fi
My test result if successful looks like:
> Lambda-Image-Compression#1.0.0 test
> mocha
myLambda
RUNNING OPTIMSATION
download
downloadImage: 69.381ms
End of step null
✓ Should move testImage.png from srcBucket to dstBucket and return true (286ms)
1 passing (299ms)
Failure shows the following:
TEST RESULT:
> Lambda-Image-Compression#1.0.0 test
> mocha
myLambda
RUNNING OPTIMSATION
download
1) Should move testImage.png from srcBucket to dstBucket and return true
0 passing (22ms)
1 failing
1) myLambda Should move testImage.png from srcBucket to dstBucket and return true:
ReferenceError: s3 is not defined
at download (index.js:32:4)
at nextTask (node_modules/async/dist/async.js:5273:14)
at Object.waterfall (node_modules/async/dist/async.js:5283:5)
at exports.handler (index.js:24:8)
at error (node_modules/lambda-tester/lib/runner.js:151:25)
at Promise.resolve.then (node_modules/lambda-tester/lib/runner.js:138:24)
You could do something along the lines of:
VALID="$(npm test | grep -o 'failing')"
And then:
if [[ $VALID != "failing" ]] ...
Another way would be to pick up on " 0 passing":
VALID="$(npm test | grep -o ' 0 passing')"
And for the conditional:
if [[ $VALID != " 0 passing" ]] ...
In either case if the word(s) "failing" or " 0 passing" are found in the returned string it would indicate the test was a failure. The key is to find something unique to the passing/failure output.
NOTE: Study the output carefully; The "0 passing" string is not fool proof, since you technically could have 10 tests... grep would
see the "10 passing" and mistake it for a failure. Maybe you don't
have that many tests, but definitely be aware of it, or include the
exact number of spaces that are leading up to it.
Though the question is fairly old and has been answered already, I would like to share how I did it since the "correct" solutions seems a bit error-prone.
Npm - Prebuild:
Npm allows you to chain commands in the package.json file, allowing tests to run before your build:
{
"scripts": {
"build": "<build>",
"prebuild": "npm run test",
"test": "<test>",
}
}
so when your tests fail, the build command won't be executed. If your build command is called e.g. deploy, prebuild must be renamed to predeploy. (Check npm documentation here)
Check if build was successful
in your shell script now add the following logic:
delete build folder
build
check if build folder has been created
which could look something like this:
#!/bin/bash
BUILD_DIR="./build"
rm -r $BUILD_DIR
npm run build
if [ -d "$BUILD_DIR" ]; then
echo "Build was successful! Starting Deployment";
fi
Again, if your build folder is called differently, BUILD_DIR must be adjusted accordingly.