I have a gitlab-ci.yml in which i am running a script as shown here, where i compare 2 files
./build/tests-i2c-regs < tests/test_cases.txt > build/lcd-regs-test-output.txt
cmp build/lcd-regs-test-output.txt tests/validate-lcd-regs-test.txt
if [ $? -eq 1 ]; then
exit 1
fi
and based on the exit code i have to kill the pipeline if the cmp command results in an exit-code 1
The script runs properly in my ubuntu20.04 machine where cmp shows no difference between the files being compared but in the pipeline a result popped out and build fails
Inside the pipeline:
$ bash tests/run-lcd-output-test.sh
build/lcd-display-output.txt tests/validate-lcd-test.txt differ: char 4106, line 101
Cleaning up project directory and file based variables 00:03
ERROR: Job failed: exit code 1
In my ubuntu20.04 machine
user#zs-l-20 /tmp/build-exit/ci-for-lcd-keypad $ cmp build/lcd-display-output.txt tests/validate-lcd-test.txt
user#zs-l-20 /tmp/build-exit/ci-for-lcd-keypad $ echo $?
0
This is the image used and before_script
image: debian:bullseye-slim
before_script:
- apt update
- apt install wget -y
- apt-get update
- apt install build-essential make perl -y
- wget https://raw.githubusercontent.com/torvalds/linux/master/scripts/checkpatch.pl
- chmod +x checkpatch.pl
Can someone explain why cmp behave differently in a ci/cd piepeline
Related
I am creating a lambda function from a docker image, this docker image actually runs a bash script inside of the docker container but when I tried to test that then it gives this following error. But this is successful in local. I tested with commented and uncommented entrypoint. Please help me to figure it out.
The dockerfile -
FROM amazon/aws-cli
USER root
ENV AWS_ACCESS_KEY_ID XXXXXXXXXXXXX
ENV AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXX
ENV AWS_DEFAULT_REGION ap-south-1
# RUN mkdir /tmp
COPY main.sh /tmp
WORKDIR /tmp
RUN chmod +x main.sh
RUN touch file_path_final.txt
RUN touch file_path_initial.txt
RUN touch output_final.json
RUN touch output_initial.json
RUN chmod 777 file_path_final.txt
RUN chmod 777 file_path_initial.txt
RUN chmod 777 output_final.json
RUN chmod 777 output_initial.json
RUN yum install jq -y
# ENTRYPOINT ./main.sh ; /bin/bash
ENTRYPOINT ["/bin/sh", "-c" , "ls && ./tmp/main.sh"]
The error -
START RequestId: 8d689260-e500-45d7-aac8-ae260834ed96 Version: $LATEST
/bin/sh: ./tmp/main.sh: No such file or directory
/bin/sh: ./tmp/main.sh: No such file or directory
END RequestId: 8d689260-e500-45d7-aac8-ae260834ed96
REPORT RequestId: 8d689260-e500-45d7-aac8-ae260834ed96 Duration: 58.29 ms Billed Duration: 59 ms Memory Size: 128 MB Max Memory Used: 3 MB
RequestId: 8d689260-e500-45d7-aac8-ae260834ed96 Error: Runtime exited with error: exit status 127
Runtime.ExitError
Here how i did it to Run A C++ over a bash script :
#Pulling the node image from the AWS WCR PUBLIC docker hub.
FROM public.ecr.aws/lambda/provided:al2.2022.10.11.10
#Setting the working directory to /home.
WORKDIR ${LAMBDA_RUNTIME_DIR}
#Copying the contents of the current directory to the working directory.
COPY . .
#This is installing ffmpeg on the container.
RUN yum update -y
# Install sudo, wget and openssl, which is required for building CMake
RUN yum install sudo wget openssl-devel -y
# Install development tools
RUN sudo yum groupinstall "Development Tools" -y
# Download, build and install cmake
RUN yum install -y make
#RUN wget https://github.com/Kitware/CMake/releases/download/v3.22.3/cmake-3.22.3.tar.gz && tar -zxvf cmake-3.22.3.tar.gz && cd ./cmake-3.22.3 && ./bootstrap && make && sudo make install
RUN yum -y install gcc-c++ libcurl-devel cmake3 git
RUN ln -s /usr/bin/cmake3 /usr/bin/cmake
RUN ln -s /usr/bin/ctest3 /usr/bin/ctest
RUN ln -s /usr/bin/cpack3 /usr/bin/cpack
# get cmake versin
RUN cmake --version
RUN echo $(cmake --version)
#This is installing the nodejs and npm on the container.
RUN ./build.sh
RUN chmod 755 run.sh bootstrap
#This is running the nodejs application.
CMD [ "run.sh" ]
You will need a bootstrap file in the root directory : (FROM DOC)
#!/bin/sh
set -euo pipefail
# Initialization - load function handler
source $LAMBDA_RUNTIME_DIR/"$(echo $_HANDLER | cut -d. -f1).sh"
# Processing
while true
do
HEADERS="$(mktemp)"
# Get an event. The HTTP request will block until one is received
EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
# Extract request ID by scraping response headers received above
REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
# Run the handler function from the script
RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")
# Send the response
curl -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response" -d "$RESPONSE"
done
I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.
.gitlab.ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master # Run this job only on changes for stage branch
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "${shell} git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
Error: .gitlab-deploy-prod.sh: line 7: -o: command not found
How can i check if directory existing??
What i've tried.
#!/bin/bash
# Get servers list
set -f
# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN
eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"
if [-d "/flask-ci-cd" ] # check if directory exists
then
eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
eval "git pull https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi
I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.
Your script has some errors.
Do not use eval. No, eval does not work that way. eval is evil
When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
Take your time and research how quoting and word splitting works in shell.
git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
Read documentation for the utilities you use.
No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.
How can i check if directory existing??
Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[#]}" [ -d "/flask-ci-cd" ]; then
...
In case of directory name with spaces I would go with:
if "${shell[#]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then
Pass set -x to sh to see what's happening also on the remote side.
For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF
But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:
echo "Deploy project on server ${SERVER_URL}"
work() {
if [ ! -d /flask-ci-cd ]; then
# Note: git_token is expanded on host side
git clone https://sbhusal123:"${git_token}"#gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token) # transfer variables you need
$(declare -f work) # transfer function you need
work # call the function.
EOF
Updated answer for future reads.
.gitlab-ci.yml
stages:
- test
- deploy
test_app:
image: python:latest
stage: test
before_script:
- python -V
- pip install virtualenv
- virtualenv env
- source env/bin/activate
- pip install flask
script:
- cd flask-ci-cd
- python test.py
prod-deploy:
stage: deploy
only:
- master
before_script:
- mkdir -p ~/.ssh
- echo -e "$RSA_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash .gitlab-deploy-prod.sh
environment:
name: deploy
.gitlab-deploy-prod.sh
#!/bin/bash
# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN
echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
echo "\n Cloning into remote repo..."
git clone https://sbhusal123:${git_token}#gitlab.com/sbhusal123/flask-ci-cd.git
# Create and activate virtualenv
echo "\n Creating virtual env"
python3 -m venv env
else
echo "Pulling remote repo origin..."
cd flask-ci-cd
git pull
cd ..
fi
# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate
# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF
There is a test command which is explicit about checking files and directories:
test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands
Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)
******** UPDATE *********
Bash script has no errors, checked with https://www.shellcheck.net/
Adding to the Dockerfilethe line
RUN tty | sed -e "s:/dev/::"
Outputs:
No tty
Next line on Dockerfile always fails:
ENTRYPOINT ["/usr/local/bin/jenkins.sh"]
I leave an image in order to clarify. In short, I think I need to attach a tty in some way to the batch script, but dunno how to do it.
Thanks
------------------- OLD CONTENT -------------------
I need to update a Jenkins image to 2.138.2. An excerpt of the original Dockerfile is as follows:
FROM openjdk:8-jdk
RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/lists/*
# ...
# Use tini as subreaper in Docker container to adopt zombie processes
COPY tini_pub.gpg ${JENKINS_HOME}/tini_pub.gpg
RUN curl -fsSL https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-$(dpkg --print-architecture) -o /sbin/tini \
&& curl -fsSL https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-$(dpkg --print-architecture).asc -o /sbin/tini.asc \
&& gpg --import ${JENKINS_HOME}/tini_pub.gpg \
&& gpg --verify /sbin/tini.asc \
&& rm -rf /sbin/tini.asc /root/.gnupg \
# ...
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/jenkins.sh"]
Using this Dockerfile FAILS due to gpg --import statement now needs to be fixed using --no-tty option. So that line remains as follows:
&& gpg --no-tty --import ${JENKINS_HOME}/tini_pub.gpg \
That's not fine since the execution of jenkins.sh now fails in several ways. The code of the script starts as follows:
#! /bin/bash -e
: "${JENKINS_WAR:="/usr/share/jenkins/jenkins.sh
This script is called from the Dockerfile in this line:
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/jenkins.sh"]
But now fails with several errors and seems to be impossible to process the file, nor removing the shebang line, nor removing the '-' or the '-e' option. The rest of the file is not processed fine if we change bash to other shell (not odd) nor removing the -e option (if I do that, the entrypoint does not find the jenkins.sh script).
Sumarizing, I've needed to remove a tty from gpg but doing that, I've lost access to bash scripting.
I've checked about the applied workaround, the workaround is described here ( (if I'm right, case is number 8, gpg might write to the tty at some point):
https://lists.gnupg.org/pipermail/gnupg-users/2017-April/058162.html
Is there any way to attach a tty to the entrypointor having any settings in the script in order to allow this work fine?
Thanks.
Finally runned on a Linux VM and no problems. Running it on Windows is the problem.
On my computer bash starts and lasts on docker:git.
~ # bash
bash-4.4# ps
PID USER TIME COMMAND
1 root 0:00 sh
26 root 0:00 bash
32 root 0:00 ps
bash-4.4# echo $0
bash
bash-4.4# echo $SHELL
/bin/ash
ash seems a bit uneasy but I'm able to run #!/bin/bash file so it's fine so far.
However, on Gitlab CI on gitlab.com, command bash doesn't return anything but it doesn't seem to keep running. Why is this?
$ apk add --update bash
$ bash
$ ps && pwd && echo $0 && echo $SHELL && ls /bin
PID USER TIME COMMAND
1 root 0:00 /bin/sh
10 root 0:00 /bin/sh
25 root 0:00 ps
/builds/230s/industrial_calibration
/bin/sh
ash
base64
bash
bashbug
:
More detailed output on my computer:
$ lsb_release -a|grep Description
No LSB modules are available.
Description: Ubuntu 16.04.4 LTS
$ docker pull docker:git
$ docker images | grep -i docker
docker git 5c58d1939c5d 10 days ago 152MB
$ docker run -it docker:git
~ # apk add --update bash
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
:
(10/10) Installing tar (1.29-r1)
Executing busybox-1.27.2-r11.trigger
OK: 37 MiB in 31 packages
~ # bash
bash-4.4# ps
PID USER TIME COMMAND
1 root 0:00 sh
26 root 0:00 bash
32 root 0:00 ps
bash-4.4# echo $0
bash
bash-4.4# echo $SHELL
/bin/ash
.gitlab-ci.yml used (this fails at the last line as the passed file uses bash specific syntax):
image: docker:git
before_script:
- apk add --update bash coreutils tar # install industrial_ci depedencies
- bash
- git clone https://github.com/plusone-robotics/industrial_ci.git .ci_config -b gitlab_modularize
- ps && pwd && echo $0 && echo $SHELL && ls /bin
- source ./.ci_config/industrial_ci/src/tests/gitlab_module.sh
UPDATE: Sourcing bash-based file in bash -c indeed works, but it's probably not useful to me as what I really want is to use a function defined in that file and because bash -c line terminates and doesn't carry the context, the function won't be available in the later lines IMO.
- /bin/bash -c "source ./.ci_config/industrial_ci/src/tests/gitlab_module.sh"
- pass_sshkey_docker
image: alpine
before_script:
- apk add --update bash coreutils tar
- bash
- echo smth
Now imagine your are the computer. You wait for each command to finish before executing the next one, you don't use the keyboard. So what do you do? Let's try it with alpine, substituting newline with ;:
$ docker run -ti --rm alpine sh -c 'apk add --update bash; bash; echo smth'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/6) Installing pkgconf (1.3.10-r0)
(2/6) Installing ncurses-terminfo-base (6.0_p20171125-r0)
(3/6) Installing ncurses-terminfo (6.0_p20171125-r0)
(4/6) Installing ncurses-libs (6.0_p20171125-r0)
(5/6) Installing readline (7.0.003-r0)
(6/6) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
Executing busybox-1.27.2-r7.trigger
OK: 13 MiB in 17 packages
bash-4.4#
YOU DON'T TOUCH THE KEYBOARD. You can wait endlessly for the bash-4.4# line to disapear as bash will wait endlessly for you to type anything. The command echo smth will never execute, gitlab will timeout waiting for bash to end, the end.
Now if you want to execute something in alpine using bash using gitlab-ci i suggest doing it this way: create a executable script ci-myscript.sh that you git add&commit to your repo:
$ cat ci-myscript.sh
#!/bin/bash
git clone https://github.com/plusone-robotics/industrial_ci.git .ci_config -b gitlab_modularize
ps && pwd && echo $0 && echo $SHELL && ls /bin
source ./.ci_config/industrial_ci/src/tests/gitlab_module.sh
The first line #!/bin/bash tells that shell should execute this script under bash. Now from your gitlab-ci you run:
image: docker:git
before_script:
- apk add --update bash coreutils tar
- ./ci-myscript-sh
Creating such scripts is actually a good workflow, because you can test the script locally on your computer before testing it in gitlab-ci.
The other option is to single call bash as suggested by #Mazel in comments:
image: docker:git
before_script:
- apk add --update bash coreutils tar
- bash -c 'git clone https://github.com/plusone-robotics/industrial_ci.git .ci_config -b gitlab_modularize; ps && pwd && echo $0 && echo $SHELL && ls /bin; source ./.ci_config/industrial_ci/src/tests/gitlab_module.sh'
That way you need to call everything in single line, because the next line won't have the same enviroment as the previous line.
I have a deploy script I only want to run if my test is successful but believe there is an issue with my conditional statement if [ "$VALID" ]
#!/bin/bash
# install dependencies
echo 'INSTALLING YARN'
npm install yarn -g
echo "INSTALLING DEPENDENCIES"
yarn install
echo "TESTING"
VALID="$(npm test)"
if [ "$VALID" ]
then
# ZIP up the code
echo 'INSTALLING ZIP'
apt-get update
echo "y" | apt-get install zip
echo 'ZIPPING'
zip -r ./Lambda-Image-Compression.zip index.js node_modules
# install aws cli so we can deploy code
echo 'INSTALLING PIP'
# echo "y" | apt-get install python-pip
echo "y" | apt-get install python-pip python-dev build-essential
echo "y" | pip install --upgrade pip
# echo "y" | sudo pip install --upgrade virtualenv
echo 'INSTALLING AWSCLI'
pip install awscli
# Copy config file to root so AWS config & credentials are set
echo 'MAKING AWS CREDENTIALS'
CREDENTIALS="[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY"
touch .aws/credentials
"$CREDENTIALS" > .aws/credentials
echo "MOVING AWS CONFIG"
cp -R .aws ~/.
# Upload to AWS
echo 'UPDATING LAMBDA FUNCTION'
aws lambda update-function-code \
--function-name resizeHandler \
--zip-file fileb://Lambda-Image-Compression.zip \
--region ap-southeast-2
fi
My test result if successful looks like:
> Lambda-Image-Compression#1.0.0 test
> mocha
myLambda
RUNNING OPTIMSATION
download
downloadImage: 69.381ms
End of step null
✓ Should move testImage.png from srcBucket to dstBucket and return true (286ms)
1 passing (299ms)
Failure shows the following:
TEST RESULT:
> Lambda-Image-Compression#1.0.0 test
> mocha
myLambda
RUNNING OPTIMSATION
download
1) Should move testImage.png from srcBucket to dstBucket and return true
0 passing (22ms)
1 failing
1) myLambda Should move testImage.png from srcBucket to dstBucket and return true:
ReferenceError: s3 is not defined
at download (index.js:32:4)
at nextTask (node_modules/async/dist/async.js:5273:14)
at Object.waterfall (node_modules/async/dist/async.js:5283:5)
at exports.handler (index.js:24:8)
at error (node_modules/lambda-tester/lib/runner.js:151:25)
at Promise.resolve.then (node_modules/lambda-tester/lib/runner.js:138:24)
You could do something along the lines of:
VALID="$(npm test | grep -o 'failing')"
And then:
if [[ $VALID != "failing" ]] ...
Another way would be to pick up on " 0 passing":
VALID="$(npm test | grep -o ' 0 passing')"
And for the conditional:
if [[ $VALID != " 0 passing" ]] ...
In either case if the word(s) "failing" or " 0 passing" are found in the returned string it would indicate the test was a failure. The key is to find something unique to the passing/failure output.
NOTE: Study the output carefully; The "0 passing" string is not fool proof, since you technically could have 10 tests... grep would
see the "10 passing" and mistake it for a failure. Maybe you don't
have that many tests, but definitely be aware of it, or include the
exact number of spaces that are leading up to it.
Though the question is fairly old and has been answered already, I would like to share how I did it since the "correct" solutions seems a bit error-prone.
Npm - Prebuild:
Npm allows you to chain commands in the package.json file, allowing tests to run before your build:
{
"scripts": {
"build": "<build>",
"prebuild": "npm run test",
"test": "<test>",
}
}
so when your tests fail, the build command won't be executed. If your build command is called e.g. deploy, prebuild must be renamed to predeploy. (Check npm documentation here)
Check if build was successful
in your shell script now add the following logic:
delete build folder
build
check if build folder has been created
which could look something like this:
#!/bin/bash
BUILD_DIR="./build"
rm -r $BUILD_DIR
npm run build
if [ -d "$BUILD_DIR" ]; then
echo "Build was successful! Starting Deployment";
fi
Again, if your build folder is called differently, BUILD_DIR must be adjusted accordingly.