Nested stages in CI jobs - continuous-integration

Question

In GitLab job description it is possible to specify stages, where jobs will be grouped by stages and executed in parallel. Imagine that I'd like to do the following:
Build a release binary.
Build a release Docker image for release binary.
Build a debug binary.
Build a debug Docker image for debug binary.
With no nested stages, I can try building release and debug binaries at the same time, and later build both images. But, this is terribly inefficient because one of the builds takes a lot longer than the other, yet, I cannot start creating an image for the build that finished first.
If only it was possible to arrange for the Docker image building job to start as soon as either the first build finished, it would be perfect. One way this might have been possible is if I could specify nested stages, where, say, stage build-all had two nested stages: build-release and build-debug, each composed of two jobs: build-release-binary, build-release-image, and, similarly, build-debug-binary, build-debug-image.
Since I'm new to GitLab, I would also appreciate a negative answer, i.e. knowing that it is not possible is also useful.

Answer 1

Problem
To first confirm your problem, I imagine you have a setup like this:
.gitlab-ci.yml:
stages:
- build-binaries
- build-images
# Binaries
build-release-binary:
stage: build-binaries
script:
- make release
build-debug-binary:
stage: build-binaries
script:
- make debug
# Docker Images
build-release-image:
stage: build-images
dependencies:
- build-release-binary
script:
- docker build -t wvxvw:release .
build-debug-image:
stage: build-images
dependencies:
- build-debug-binary
script:
- docker build -t wvxvw:debug .
And that should produce a pipeline like this:
build-binaries build-images
______________________ _____________________
| | | |
| build-release-binary |----+--+--->| build-release-image |
|______________________| / \ |_____________________|
| |
______________________ | | _____________________
| | | | | |
| build-debug-binary |---/ \-->| build-debug-image |
|______________________| |_____________________|
Assessment
You are correct that no jobs from the build-images stage will begin until all jobs from the build-binaries stage complete (even though the job's dependencies are met).
There is a GitLab issue open that discusses this:
gitlab-org/gitlab-ce#49964: Allow running a CI job if its dependencies succeeded
I've added a comment pointing out the improvements that could be made in this case. In the future, the pipeline might then look like this (note the separate connecting lines):
build-binaries build-images
______________________ _____________________
| | | |
| build-release-binary |----------->| build-release-image |
|______________________| |_____________________|
______________________ _____________________
| | | |
| build-debug-binary |----------->| build-debug-image |
|______________________| |_____________________|
Workaround
Sometimes if you have sequential tasks, it's easier to simply run them in a single job. This avoids the overhead of firing up another job, when you already have everything ready to go in the first job.
As a work-around, you could simply flatten your pipeline into a single stage which would build both the binary and the Docker image:
.gitlab-ci.yml:
stages:
- build
build-release:
stage: build
script:
- make release
- docker build -t wvxvw:release .
build-debug:
stage: build
script:
- make debug
- docker build -t wvxvw:debug .
Your pipeline would then of course look like this:
build
_______________
| |
| build-release |
|_______________|
_______________
| |
| build-debug |
|_______________|
I've worked with a team to simplify their pipeline in a similar manner, and we were pleased with the results.

Answer 2

As of Gitlab 12.2, this was fixed with the needs clause, so arbitrary DAGs are now allowed. You can visualize the graph as of Gitlab 13.1 (Beta).
For example, imagine you want to run pylint and unit tests in parallel, and then check the coverage of your unit tests, but without waiting for pylint to finish.
stages:
- Checks
- SecondaryChecks
pylint:
stage: Checks
script: pylint
unittests:
stage: Checks
script: coverage run -m pytest -rs --verbose
testcoverage:
stage: SecondaryChecks
needs: ["unittests"]
script: coverage report -m | grep -q "TOTAL.*100%"
Note that 'needs' only works for targets defined in previous stages. Hence the need for two stages here.