Howto lint a bash script inside a YAML file - bash

Question

I am trying to create a GitLab CI/CD job that takes a bash script from a YAML file and checks if the syntax is correct.
I have a YAML file called .gitlab-ci.template.yml with the following content:
image: node:10.15.3-alpine
stages:
- deploy
deploy:
stage: deploy
script:
- apk_install
- yarn_install
- build
- deploy
.bash: &bash |
function apk_install() {
apk add jq git curl grep
}
function yarn_install() {
yarn
}
function build() {
echo "Build"
if [ -f ".gitlab-ci.template.yml" ]; then
echo "Just a dumb test for more lines"
fi
}
function deploy() {
echo "Deploy"
}
before_script:
- *bash
I would like to take the bash part and test it. I installed shyaml to get values from the YAML file, like so:
FILE=`cat .gitlab-ci.template.yml | shyaml get-value before_script`
The contents of $FILE would be:
- "function apk_install() {\n apk add jq git curl grep\n}\n\nfunction yarn_install()\ \ {\n yarn\n}\nfunction build() {\n echo \"Build\"\n \n if [ -f \".gitlab-ci.template.yml\"\ \ ]; then\n echo \"Just a dumb test for more lines\"\n fi\n}\n\nfunction deploy()\ \ {\n echo \"Deploy\"\n}\n"
Then with some the following command I try to get a valid file again:
echo $FILE | xargs -0 printf '%b\n' | sed 's/\\[[:space:]]\\[[:space:]]/ /g' | sed 's/\\"/"/g' | sed 's/\\[[:space:]]//g'
Now I can remove some chars from the beginning and end. But I was wondering would there be any better/easier way?

Answer 1

Have you considered using yq?
Here's an expression to get the shell script out of your file:
yq -r '.[".bash"]' .gitlab-ci.template.yml
The result is the actual script. Now, you just need to pipe it to a bash linter. I looked through this site for you, but I couldn't find the bash syntax parser that is often sited in the bash posts (YMMV). I did find ShellChecker via google, but I didn't evaluate thoroughly (so, use as you see fit).
Ultimately, your code may look like this:
yq -r '.[".bash"]' .gitlab-ci.template.yml | ShellChecker # or use your favorite bash linter

Answer 2

To lint shell script embedded in GitLab CI YAML, a combination of git, yq, and shellcheck does the trick as demonstrated by this GitLab CI job:
shellcheck:
image: alpine:3.15.0
script:
- |
# shellcheck shell=sh
# Check *.sh
git ls-files --exclude='*.sh' --ignored -c -z | xargs -0r shellcheck -x
# Check files with a shell shebang
git ls-files -c | while IFS= read -r file; do
if head -n1 "${file}" |grep -q "^#\\! \?/.\+\(ba|d|k\)\?sh" ; then
shellcheck -x "${file}"
fi
done
# Check script embedded in GitLab CI YAML named *.gitlab-ci.yml
newline="$(printf '\nq')"
newline=${newline%q}
git ls-files --exclude='*.gitlab-ci.yml' --ignored -c | while IFS= read -r file; do
yq eval '.[] | select(tag=="!!map") | (.before_script,.script,.after_script) | select(. != null ) | path | ".[\"" + join("\"].[\"") + "\"]"' "${file}" | while IFS= read -r selector; do
script=$(yq eval "${selector} | join(\"${newline}\")" "${file}")
if ! printf '%s' "${script}" | shellcheck -; then
>&2 printf "\nError in %s in the script specified in %s:\n%s\n" "${file}" "${selector}" "${script}"
exit 1
fi
done
done
The job will check script, after_script, and before_script which are all the places where shell script can appear in GitLab CI YAML. It searches for GitLab CI YAML files matching the naming convention *.gitlab-ci.yml.
For convenience, this job also checks *.sh and files that start with a shell shebang.
I also posted this information to my site at https://candrews.integralblue.com/2022/02/shellcheck-scripts-embedded-in-gitlab-ci-yaml/

Answer 3

Just noticed I should get the first occurence from before_script with shyaml. So I changed:
FILE=`cat .gitlab-ci.template.yml | shyaml get-value before_script`
to:
FILE=`cat .gitlab-ci.template.yml | shyaml get-value before_script.0`
Which will get the script as-is.