how do i run loops simultaneously in gitlab-ci? - bash

Question

I have the following script in my gitlab-ci and will like to run the loops same time, anyone knows a great way to do this? so that they both run at same time
NOTE the job is a manual job and am looking for a single button click to loop through all the packages in the bash script as shown below
when: manual
script:
- |-
for PACKAGE in name1 name2; do
export IMAGE="$CI_REGISTRY/$GITLAB_REPO/$PACKAGE:${BUILD_TAG}"
docker build -t ${IMAGE} -f $PACKAGE/Dockerfile .
docker push ${IMAGE}
done
currently it runs first for name1 and then after that is finished then runs for name2. I will like to run both at same exact time since there is no dependency
Here is what i tried from an answer on SO => (https://unix.stackexchange.com/a/216475/138406)
when: manual
script:
- |-
task(){
export IMAGE="$CI_REGISTRY/$GITLAB_REPO/$1:${BUILD_TAG}"
docker build -t ${IMAGE} -f $1/Dockerfile .
docker push ${IMAGE}
}
for PACKAGE in name1 name2; do
task "$PACKAGE" &
done
This works in regular bash script but when i used it with gitlab-ci, it doesnt run as expected and does not even run any of the commands and just succeeds the job instantly
Anyone willing to help on where the issue is and how to solve this issue?

Answer 1

To achieve your use case, I'd suggest you rather "parallelize" the build using several dedicated GitLab-CI jobs, rather than using several bash jobs in a single GitLab-CI job.
Proof-of-concept:
stages:
- push
.push-template:
stage: push
image: docker:latest
services:
- docker:dind
variables:
IMAGE: "${CI_REGISTRY}/${GITLAB_REPO}/${PACKAGE}:${BUILD_TAG}"
# where ${PACKAGE} should be provided by the child jobs...
before_script: |
if [ -z "${PACKAGE}" ]; then
echo 'Error: variable PACKAGE is undefined' >&2
false
fi
# just for completeness, this may be required:
echo "${CI_JOB_TOKEN}" | docker login -u "${CI_REGISTRY_USER}" --password-stdin "${CI_REGISTRY}"
script:
- docker build -t "${IMAGE}" -f "${PACKAGE}/Dockerfile" .
- docker push "${IMAGE}"
- docker logout "${CI_REGISTRY}" || true
push-name1:
extends: .push-template
variables:
PACKAGE: name1
push-name2:
extends: .push-template
variables:
PACKAGE: name2
See the .gitlab-ci.yml reference manual for details on the extends keyword.

Answer 2

just succeeds the job instantly
That is what running in the background means - it means that the main process will continue instantly. You have to wait for background processes to finish.
- |-
task(){
export IMAGE="$CI_REGISTRY/$GITLAB_REPO/$1:${BUILD_TAG}"
docker build -t ${IMAGE} -f $1/Dockerfile .
docker push ${IMAGE}
}
for PACKAGE in name1 name2; do
task "$PACKAGE" &
done
wait
But that will not catch any errors, which will result in problems going undetected. You would have to collect pids and wait on the individually:
...
childs=""
for package in name1 name2; do
task "$package" &
childs="$childs $!"
done
for pid in $childs; do
if ! wait "$pid"; then
echo "Process with pid=$pid failed"
kill $childs
wait
exit 1
fi
done
But anyway, this is cumbersome and it's reinventing the wheel. Install GNU xargs (or even better parallel) and make sure your docker container has bash shell. Then, just export the function and run it in subprocess with xargs:
...
export -f task
printf "%s\n" name1 name2 | xargs -P0 -d '\n' bash -xeuo pipefail -c 'func "$#"' --
You may want to research https://man7.org/linux/man-pages/man1/xargs.1.html or even https://www.gnu.org/software/bash/manual/html_node/Job-Control-Basics.html .
Definitely, instead of writing long scripts in .gitlab-ci yaml file - move it all a dedicated script file, so that you can test the run locally. Check your scripts with shellcheck. And anyway - using docker-compose might also be simpler.
Anyway, this is all odd and for sure I wouldn't do it that way anyway. Gitlab-runner is already the tool that gives you parallelization - it runs multiple jobs in the same stage in parallel. Just run two tasks.
.todo:
script:
export IMAGE="$CI_REGISTRY/$GITLAB_REPO/${CI_JOB_NAME}:${BUILD_TAG}"
docker build -t ${IMAGE} -f ${CI_JOB_NAME}/Dockerfile .
docker push ${IMAGE}
name1:
extends: .todo
name2:
extends: .todo
Such approach will give your pipeline visible indication over what specific task failed, so you wouldn't need to scroll through mangled unreadable logs of two processes running in parallel. Just one job with one task.