so first off, I'm not sure if I'm doing something wrong or if there are bugs that need to be worked out in the go-task project on GitHub.
I'm running into issues where occasionally I run into an error that looks something like this:
context canceled
task: Failed to run task "start": task: Failed to run task "upstream:common": task: Failed to run task "upstream:common:merge:variables": task: Failed to run task "upstream:common:merge:variables:subtype": fork/exec /usr/bin/jq: bad file descriptor
Some sample code that causes this issue is:
function mergePackages() {
# Merge the files
TMP="$(mktemp)"
jq --arg keywords "$(jq '.keywords[]' "$1" "$2" | jq -s '. | unique')" -s -S '.[0] * .[1] | .keywords = ($keywords | fromjson) | .' "$1" "$2" > "$TMP"
mv "$TMP" "$3"
}
for FOLDER in project-*/; do
SUBTYPE="$(echo "$FOLDER" | sed 's/project-\(.*\)\//\1/')"
mergePackages "./.common/package.hbs.json" "project-$SUBTYPE/package.hbs.json" "./.common-$SUBTYPE/package.hbs.json" &
done
wait
The issue seems to occur most often when I'm adding & to the end of a bash command for an inline function to execute although I have seem it happen even if a function is called without &.
Is there anything I am doing wrong?
Related
I have a step in a Github Actions job:
- name: Check for changes
run: |
diff=$( git diff --name-only 'origin/main' )
changed_files=$( echo $diff | grep -c src/my_folder ) # this fails
# more script lines
# that are unrelated
This fails with Error: Process completed with exit code 1. only if grep finds nothing.
If there are matches in $diff, then this step works as intended. But of course it also needs to work without matches.
I can run this locally or inside a script without a problem, exit code is always 0 (on a Mac).
I fail to understand what the problem is. After some hours of trial and error and research I learned that apparently grep is tricky in Github actions, but I found no hint or proper documentation how I am supposed to solve this exact case.
If I change my failing line to
echo $( echo $diff | grep -c src/my_folder ) # this works and prints out the result
this gets executed without problems.
But how do I get my grep output into my variable even when there are no findings?
According to the doc, by default Github Actions enables set -e to run the step's commands. This is why an intermediate command failure may cause the whole step to fail. To take full control of your step's commands you can do like this:
- name: Check for changes
shell: bash {0}
run: |
diff=$( git diff --name-only 'origin/main' )
changed_files=$( echo $diff | grep -c src/my_folder )
# ...
# ...
Or you can just disable the default set -e at the beginning of the step's script:
- name: Check for changes
run: |
set +x
diff=$( git diff --name-only 'origin/main' )
changed_files=$( echo $diff | grep -c src/my_folder )
# ...
# ...
I would suggest to use the dorny/paths-filter action, like:
- uses: dorny/paths-filter#v2
id: changes
with:
filters: |
src:
- 'src/my_folder/**'
# run only if some file in 'src/my_folder' folder was changed
- if: steps.changes.outputs.src == 'true'
run: ...
Check the output section for details about changes
I don't know what problem grep has in Github actions but you can try something like :
...
changed_files=$( [ ! -e $diff ] && { echo $diff | grep -c src/my_folder } || echo 0 )
...
This way grep wouldn't run if $diff is empty. It would just store 0 in $changed_files
I want to write a while loop in a GitLab CI file and here is the syntax that I've tried but seems to not be working.
Is the while loop authorized in GitLab or YAML files? Or are there other ways to write it?
Here is where I used it:
- while ($(curl -X GET ${URL} | jq -r '.task.status') != "SUCCESS")
ANALYSIS_ID=$(curl -X GET ${URL} | jq -r '.task.analysisId')
Why don't you write yourself a shell/python/whatever script and just run it from the CI?
YAML is not the suitable language to perform such a things (e.g. while loops, large conditions, for loops) and should not be used that way...
So I did this to resolve my issue , its to create a script in which I ve wrote the loop while and this script return the value that I needed, and then I called this script in my gitlab_ci file as below :
- ANALYSIS_ID=$(**./checkUrl.sh** $URL)
And if needed as an example the script that I used
#!/bin/bash
success="SUCCESS"
condition="$(curl -X GET "$1" | jq -r '.task.status')"
while [ "$condition" != "$success" ]
do
ANALYSIS_Id="$(curl -X GET "$1" | jq -r '.task.analysisId')"
done
return "$ANALYSIS_Id"
I am trying to create a GitLab CI/CD job that takes a bash script from a YAML file and checks if the syntax is correct.
I have a YAML file called .gitlab-ci.template.yml with the following content:
image: node:10.15.3-alpine
stages:
- deploy
deploy:
stage: deploy
script:
- apk_install
- yarn_install
- build
- deploy
.bash: &bash |
function apk_install() {
apk add jq git curl grep
}
function yarn_install() {
yarn
}
function build() {
echo "Build"
if [ -f ".gitlab-ci.template.yml" ]; then
echo "Just a dumb test for more lines"
fi
}
function deploy() {
echo "Deploy"
}
before_script:
- *bash
I would like to take the bash part and test it. I installed shyaml to get values from the YAML file, like so:
FILE=`cat .gitlab-ci.template.yml | shyaml get-value before_script`
The contents of $FILE would be:
- "function apk_install() {\n apk add jq git curl grep\n}\n\nfunction yarn_install()\ \ {\n yarn\n}\nfunction build() {\n echo \"Build\"\n \n if [ -f \".gitlab-ci.template.yml\"\ \ ]; then\n echo \"Just a dumb test for more lines\"\n fi\n}\n\nfunction deploy()\ \ {\n echo \"Deploy\"\n}\n"
Then with some the following command I try to get a valid file again:
echo $FILE | xargs -0 printf '%b\n' | sed 's/\\[[:space:]]\\[[:space:]]/ /g' | sed 's/\\"/"/g' | sed 's/\\[[:space:]]//g'
Now I can remove some chars from the beginning and end. But I was wondering would there be any better/easier way?
Have you considered using yq?
Here's an expression to get the shell script out of your file:
yq -r '.[".bash"]' .gitlab-ci.template.yml
The result is the actual script. Now, you just need to pipe it to a bash linter. I looked through this site for you, but I couldn't find the bash syntax parser that is often sited in the bash posts (YMMV). I did find ShellChecker via google, but I didn't evaluate thoroughly (so, use as you see fit).
Ultimately, your code may look like this:
yq -r '.[".bash"]' .gitlab-ci.template.yml | ShellChecker # or use your favorite bash linter
To lint shell script embedded in GitLab CI YAML, a combination of git, yq, and shellcheck does the trick as demonstrated by this GitLab CI job:
shellcheck:
image: alpine:3.15.0
script:
- |
# shellcheck shell=sh
# Check *.sh
git ls-files --exclude='*.sh' --ignored -c -z | xargs -0r shellcheck -x
# Check files with a shell shebang
git ls-files -c | while IFS= read -r file; do
if head -n1 "${file}" |grep -q "^#\\! \?/.\+\(ba|d|k\)\?sh" ; then
shellcheck -x "${file}"
fi
done
# Check script embedded in GitLab CI YAML named *.gitlab-ci.yml
newline="$(printf '\nq')"
newline=${newline%q}
git ls-files --exclude='*.gitlab-ci.yml' --ignored -c | while IFS= read -r file; do
yq eval '.[] | select(tag=="!!map") | (.before_script,.script,.after_script) | select(. != null ) | path | ".[\"" + join("\"].[\"") + "\"]"' "${file}" | while IFS= read -r selector; do
script=$(yq eval "${selector} | join(\"${newline}\")" "${file}")
if ! printf '%s' "${script}" | shellcheck -; then
>&2 printf "\nError in %s in the script specified in %s:\n%s\n" "${file}" "${selector}" "${script}"
exit 1
fi
done
done
The job will check script, after_script, and before_script which are all the places where shell script can appear in GitLab CI YAML. It searches for GitLab CI YAML files matching the naming convention *.gitlab-ci.yml.
For convenience, this job also checks *.sh and files that start with a shell shebang.
I also posted this information to my site at https://candrews.integralblue.com/2022/02/shellcheck-scripts-embedded-in-gitlab-ci-yaml/
Just noticed I should get the first occurence from before_script with shyaml. So I changed:
FILE=`cat .gitlab-ci.template.yml | shyaml get-value before_script`
to:
FILE=`cat .gitlab-ci.template.yml | shyaml get-value before_script.0`
Which will get the script as-is.
I have the following script. Unfortunately I not able to get it run on Jenkins.
#!/bin/bash
function pushImage () {
local serviceName=$1
local version=$(getImageVersionTag $serviceName)
cd ./dist/$serviceName
docker build -t $serviceName .
docker tag $serviceName gcr.io/$PROJECT_ID/$serviceName:$version
docker tag $serviceName gcr.io/$PROJECT_ID/$serviceName:latest
docker push gcr.io/$PROJECT_ID/$serviceName
cd ../..
}
function getImageVersionTag () {
local serviceName=$1
if [ $BUILD_ENV = "dev" ];
then
echo $(timestamp)
else
if [ $serviceName = "api" ];
then
echo $(git tag -l --sort=v:refname | tail -1 | awk -F. '{print $1"-"$2"-"$3"-"$4}')
else
echo $(git tag -l --sort=refname | tail -1 | awk -F. '{print $1"-"$2"-"$3"-"$4}')
fi
fi
}
function timestamp () {
echo $(date +%s%3N)
}
set -x
## might be api or static-content
pushImage $1
I'm receiving this error on Jenkins
10:10:17 + sh push-image.sh api
10:10:17 push-image.sh: 2: push-image.sh: Syntax error: "(" unexpected
I already configured Jenkins global parameter to /bin/bash as default shell execute environment, but still having same error.
The main issue here in usage of functions, as other scripts that has been executed successfully don't have any.
How this can be fixed?
Short answer: make sure you're running bash and not sh
Long answer: sh (which is run here despite your effort of adding a shebang) is the bourne shell and does not understand the function keyword. Simply removing it will solve your issue.
Please note however that all your variable expansions should be quoted to prevent against word splitting and globbing. Ex: local version=$(getImageVersionTag "$serviceName")
See shellcheck.net for more problems appearing in your file (usage of local var=$(...)) and explicit list of snippets which are missing quotes.
I'm trying to use Jenkins to create a special git repository. I created a free-style project that just executes a shell script. When I execute this script by hand, without Jenkins, it works just fine.
From Jenkins, however, it behaves quite differently.
# this will remove all subtrees
git log | grep git-subtree-dir | tr -d ' ' | cut -d ":" -f2 | sort | uniq | xargs -I {} bash -c 'if [ -d $(git rev-parse --show-toplevel)/{} ] ; then rm -rf {}; fi'
rm -rf .git
If this part is executed by Jenkins, in console output I see this kind of errors:
rm: cannot remove '.git/objects/pack/pack-022eb85d38a41e66ad3f43a5f28809a5a3ee4a0f.pack': Device or resource busy
rm: cannot remove '.git/objects/pack/pack-05630eb059838f149ad30483bd48d37f9a629c70.pack': Device or resource busy
rm: cannot remove '.git/objects/pack/pack-26f510b5a2d15ba9372cf0a89628d743811e3bb2.pack': Device or resource busy
rm: cannot remove '.git/objects/pack/pack-33d276d82226c201eedd419e5fd24b6b906d4c03.pack': Device or resource busy
I modified this part of the script like this:
while true
do
if rm -rf .git ; then
break
else
continue
fi
done
But this doesn't help. In the task manager I see a git process that just doesn't terminate.
I conjured said script by a lot of googling and I do not understand very good what's going on.
Jenkins runs on Windows Server 2012 behind IIS; shell scripts are executed by bash shipped with git for Windows.
1/ Ensure your path is correct, and no quote/double quote escaping occurs in the process of jenkins job starting.
2/ Your command line is a bit too handy to be correctly and safely evaluated.
Put your commands in a regular script, starting with #!/bin/bash instead of thru the command line.
xargs -I {} bash -c 'if [ -d $(git rev-parse --show-toplevel)/{} ] ; then rm -rf {}; fi'
becames
xargs -I {} /path/myscript.sh {}
with
#!/bin/bash
rev-parse="$(git rev-parse --show-toplevel)"
wait
if [ -d ${rev-parse}/${1} ] ; then
rm -rf ${1}
fi
Please note that your script is really unsafe, as you rm -rf a parameter without even evaluate it before… !
3/ You can add a wait between the git and the rm to wait for the end of the git process
4/ log your git command into a log file, with a redirection >> /tmp/git-jenkins-log
5/ put all of those commands in a script (see #2)
Following is an infinite loop in case rm -rf fail
while true
do
if rm -rf .git ; then
break
else
continue
fi
done
indeed continue can be used in for or while loop to get the next entry but in this while loop it will run the same rm command forever.
Well, aparrently I was able to fix my issue by running the script from different user.
By default on Windows Jenkins executes all jobs from the user SYSTEM. I have no idea why it affects the behaviour of my script but running it with psexec from specially created user account worked.
In case anyoune is interested, I did something like this:
psexec -accepteula -h -user Jenkins -p _password_ "full/path/to/bash.exe" full/path/to/script.sh