In Azure pipeline yaml file, when defining multiple jobs in a single stage, one can specify dependencies between them. One can also specify the conditions under which each job runs.
Code #1
jobs:
- job: A
steps:
- script: echo hello
- job: B
dependsOn: A
condition: and(succeeded(), eq(variables['build.sourceBranch'], 'refs/heads/master'))
steps:
- script: echo this only runs for master
Code #2
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
Question:
Code #1 & #2 above have different orders of the dependency and condition. Does the order matters? If so, what's matter? (what's the difference between different orders)
Discuss 1 and 2 separately.
Code #1:
Since there is no data connection between job1 and job2, data connection here refers to variable sharing and etc.
So, for #1, there's no matters on order. Here you can ignore the dependsOn specified while you have no special requirements on the execution order between job A and job B.
BUT, there's one key thing you need pay attention is, the actual running order will be changed randomly when you do not specify the dependsOn. For example, most of time, they will respect with the order job A, job B. Occasionally, they will randomly run as job B, job A.
Code #2:
This must make the dependsOn specified. Because your job B is using the output variable which created/generated at job A. Since our system allow same variables name exists in different jobs, you must specify the dependsOn so that the system can know the job B should find the variable skipsubsequent from job A not others. Only this key words specified, the variables which generated in job A can be exposed and available to next jobs.
So, the nutshell is once there is any data connection between jobs, e.g variable pass, you must specify dependsOn to make the jobs has connection with each other.
Related
I have generated a file with bunch of alter statements based on certain condition on cluster using Ansible task.
Here's the sample file content
alter table test1 with throttling = 0.0;
alter table test2 with throttling = 0.0;
alter table test3 with throttling = 0.0;
I want to login and execute these ALTER statements with a delay of 2 mins. I was able to achieve this with a shell script using sleep command by copying the shell script from Control Node and execute on Remote Node.
But the problem we noticed was we were unable to check if script executed properly or failed (like authentication failed to DB, etc.)
Can we perform the same task using Ansible module and execute them one by one with some delay?
Regarding
Can we perform the same task using ansible module and execute them one by one with some delay?
the short answer, yes, of course.
A slightly longer minimal example
- name: Run queries against db test_db with pause
community.mysql.mysql_query:
login_db: test_db
query: "ALTER TABLE test{{ item }} WITH throttling = 0.0;"
loop: [1, 2, 3]
loop_control:
pause: 120 # seconds
Further Documentation
Community.Mysql - Modules
mysql_query module – Run MySQL queries
Pausing with loop
Extended loop variables
Further Q&A
How to run MySQL query in Ansible?
Ensuring a delay in an Ansible loop
I am trying to trigger a particular job in CI after either of the 2 conditions
trigger by another job in the same pipeline
OR
changes: somefile.txt
My CI is as described
job1:
stage: build
script:
- echo "JOb1"
- curl -X POST -F token=2342344444 -F "variables[TRIGGER_JOB]=job1" -F ref=master https://main.gitlab.myconmpanyxyz.com/api/v4/projects/1234/trigger/pipeline
only:
changes:
- job1.md
job2: # This does not RUN as expected because of the TRIGGER_JOB set to job1
stage: test
script:
- echo "Job2"
rules:
- if: $TRIGGER_JOB =="job2"
job3: # this RUNS as expected because of VARIABLE TRIGGER_JOB
stage: test
script:
- echo "Job3"
rules:
- if: $TRIGGER_JOB =="job1"
job4: # this also RUNS, but this should not be the expected behavior
stage: test
script:
- echo “job4“
rules:
- if: $TRIGGER_JOB == "xyz"
- changes:
- job4.md
After job1 finishes it also needs to call job4 and not any other jobs (job2 in this case). So I am using CURL to call the job itself. If there are any better ways of calling a specific job in the same CI, also please let me know.
I have already seen this stack-overflow page, but it does not help because my job needs to be triggered by either of 2 conditions which is not allowed bit gitlab-ci.
I need job4 to be called by either of the 2 conditions - if the TRIGGER_JOB=="job1" or if there are any changes in job4.md file.
Currently job4 runs if changes are made in job4.md file, however it also runs if the job1 is triggered. But afaik this should not be the expected behavior.
docs. Can anyone please give me some leads how to create this kind of design.
Your solution was almost correct, but the changes keyword with only or except does only work, if the pipeline is triggered by a push or a merge_request event. This is defined in the variable CI_PIPELINE_SOURCE. When you trigger the pipeline by calling the API, the variable CI_PIPELINE_SOURCE contains the value trigger and therefore only:changes returns always true, which triggers job1 again and ends in an endless loop. You can add a simple except rule to your job1 to prevent that:
job1:
stage: build
script:
- echo "JOb1"
- curl -X POST -F token=2342344444 -F "variables[TRIGGER_JOB]=job1" -F ref=master https://main.gitlab.myconmpanyxyz.com/api/v4/projects/1234/trigger/pipeline
only:
changes:
- job1.md
except:
variables:
- $CI_PIPELINE_SOURCE == "trigger"
You can find more information on only/except:changes in the documentation.
I have an unity ci-project.
.gitlab-ci.yml contains base .build job with one script command. Also I have multiple specified jobs for build each platform which extended base .build. I want to execute some platform-specific commands for android, so I have created separated job generate-android-apk. But if it's failing the pipeline will be failed too.(I know about allow_failure). Is it possible to extend script section between jobs without copy-pasting?
UPDATE:
since gitlab 13.9 it is possible to use !reference tags from other jobs or "templates" (which are commented jobs - using dot as prefix)
actual_job:
script:
- echo doing something
.template_job:
after_script:
- echo done with something
job_using_references_from_other_jobs:
script:
- !reference [actual_job, script]
after_script:
- !reference [.template_job, after_script]
Thanks to #amine-zaine for the update
FIRST APPROACH:
You can achieve modular script sections by utilizing 'literal blocks' (using |) like so:
.template1: &template1 |
echo install
.template2: &template2 |
echo bundle
testJob:
script:
- *template1
- *template2
See Source
ANOTHER SOLUTION:
Since GitLab 11.3 it is possible to use extend which could also work for you.
.template:
script: echo test template
stage: testStage
only:
refs:
- branches
rspec:
extends: .template1
after_script:
- echo test job
only:
variables:
- $TestVar
See Docs
More Examples
We are facing a problem where we need to run one specific job in gitlab CI. We currently not know how to solve this problem. We have multitple jobs defined in our .gitlab-ci.yml but we only need to run a single job within our pipelines. How could we just run one job e.g. job1 or job2? We can't use tags or branches as a software switch in our environment.
.gitlab-ci.yml:
before_script:
- docker info
job1:
script:
- do something
job2:
script:
- do something
You can use a gitlab variable expression with only/except like below and then pass the variable into the pipeline execution as needed.
This example defaults to running both jobs, but if passed 'true' for "firstJobOnly" it only runs the first job.
Old Approach -- (still valid as of gitlab 13.8) - only/except
variables:
firstJobOnly: 'false'
before_script:
- docker info
job1:
script:
- do something
job2:
script:
- do something
except:
variables:
- $firstJobOnly =~ /true/i
Updated Approach - rules
While the above still works, the best way to accomplish this now would be using the rules syntax. A simple example similar to my original reply is below.
If you explore the options in the rules syntax, depending on the specific project constraints there are many ways this could be achieved.
variables:
firstJobOnly: 'false'
job1:
script:
- do something
job2:
script:
- do something
rules:
- if: '$firstJobOnly == "true"'
when: never
- when: always
We faced the same problem in the past and I'm sharing with you our solution.
#Remark#
I read the answer of Jawad and I found it good and we have tried it when we faced the issue.
My remark is that adding when: manual will always show ALL your jobs in the pipeline.
So if you work in a large team, you can't prevent other collaborators to click by error or by mistake on the job you don't want to be launched.
#What I'm supposing before continuing#
Let's say that you have 4 jobs.
You need to always run (manually or automatically) job 1, job 2 and job 4 but NOT job3.
You want to only run job 3 in a specific case or just when you decide to run it.
#The idea is#
We launch the 3rd job only for tags which match a regular expression.
In the example below, it's launched for tags like helloTag.1, helloTag.2, helloTag.3... etc.
If we are in develop or master (or other branch), we will have 3 stages (stage 1, stage 2, stage 4)
Note how the 3rd job is not present in the pipeline
Go to "Repository" --> "Tags" --> "New tag"
Give the tag a name which much your regular expression
If we are in a tag having a name which starts with "helloTag.", we will have 1 stage (stage 3)
Note how other stages are not present here
#Example of .gitlab-ci file#
stages:
- myStage1
- myStage2
- myStage3
- myStage4
This is my first stage:
stage: myStage1
before_script:
- echo "my stage 1 before script"
script:
- echo "my stage 1 script"
except:
- /^helloTag.*$/
This is my second stage:
stage: myStage2
before_script:
- echo "my stage 2 before script"
script:
- echo "my stage 2 script"
except:
- /^helloTag.*$/
This is my third stage:
stage: myStage3
before_script:
- echo "my stage 3 before script"
script:
- echo "my stage 3 script"
only:
- /^helloTag.*$/
This is my fourth stage:
stage: myStage4
before_script:
- echo "my stage 4 before script"
script:
- echo "my stage 4 script"
except:
- /^helloTag.*$/
Hope that this helps you.
Simply add a when: manual to the jobs you don't want to run.
These jobs will still appear in your pipeline but won't be run, unless someone "manually" starts them through the web interface, hence the name.
Here's more info about this: https://docs.gitlab.com/ce/ci/yaml/README.html#when
If you're looking for something more "programmable", let's say run either job1 or job2 depending on a branch name or a tag, then you should have a look at the only and except keywords: https://docs.gitlab.com/ce/ci/yaml/README.html#only-and-except
> Currently it seems not to be possible with GitLab CI to have other software switches than tags or branches as provided in the other answers.
We finally switched to an other "real" CI due to too many limitations on GitLab CI. GitLab CI is unfelixble if you want to run some custom jobs in different procedures. I realy appreciated the both answers here. I'm sure they will help other users to manage this stuff. Unfortunately in our case we could not use tags, commit messages or branches as a software switch.
We are still looking for a answer on this. Feel free to give an other approach to solve this problem. I will mark the right answer once it hits. Also a bounty on this question did not result in an right answer.
The original question asks how to trigger jobs without using branch-names or tags. This leaves commit messages and environment variables as viable sources, and neither require editing your yaml file for each push.
Commit Messages
Job rules with commit message regex is a very simple and flexible solution, in my experience.
Set up your .gitlab-ci.yml like this
job1:
...
rules:
- if: '
$CI_COMMIT_MESSAGE =~ /.*run job1.*/ ||
$CI_COMMIT_MESSAGE =~ /.*run all.*/
'
job2:
...
rules:
- if: '
$CI_COMMIT_MESSAGE =~ /.*run job2.*/ ||
$CI_COMMIT_MESSAGE =~ /.*run all.*/
'
Push with a commit message like this
git commit --allow-empty -m "testing conditional job triggers for gitlab-ci based on branch names and commit messages. run job1"
git push
You'll notice this also allows you to run both jobs with a message like this
git commit --allow-empty -m "testing all jobs. run all"
git push
Environment Variables
Elsewhere in your answer to your own question you add a constraint: commit messages cannot be used. Environment variables can be set in the git cli
Set up your .gitlab-ci.yml like this
job1:
...
rules:
- if: $JOB1
job2:
...
rules:
- if: $JOB2
Push like this
git commit --allow-empty -m "triggering job1 with ci variables"
git push -o ci.variable="JOB1=anythingAtAll"
reference: https://docs.gitlab.com/ee/user/project/push_options.html
I've been trying to implement Ansible in our team to manage different kinds of application things such as configuration files for products and applications, the distribution of maintenance scripts, ...
We don't like to work with "hostnames" in our team because we have 300+ of them with meaningless names. Therefor, I started out creating aliases for them in the Ansible hosts file like:
[bpm-i]
bpm-app1-i1 ansible_user=bpmadmin ansible_host=el1001.bc
bpm-app1-i2 ansible_user=bpmadmin ansible_host=el1003.bc
[bpm-u]
bpm-app1-u1 ansible_user=bpmadmin ansible_host=el2001.bc
bpm-app1-u2 ansible_user=bpmadmin ansible_host=el2003.bc
[bpm-all:children]
bpm-i
bpm-u
Meaning we have a BPM application named "app1" and it's deployed on two hosts in integration-testing and on two hosts in user-acceptance-testing. So far so good. Now I can run an Ansible playbook to (for example) setup the SSH accesses (authorized_keys) for team members or push a maintenance script. I can run those PBs on each host seperately, on all hosts ITT or UAT or even everywhere.
But, typically, we'll have install the same application app1 again on an existing host but with a different purpose - say "training" environment. My reflex would be to do this:
[bpm-i]
bpm-app1-i1 ansible_user=bpmadmin ansible_host=el1001.bc
bpm-app1-i2 ansible_user=bpmadmin ansible_host=el1003.bc
[bpm-u]
bpm-app1-u1 ansible_user=bpmadmin ansible_host=el2001.bc
bpm-app1-u2 ansible_user=bpmadmin ansible_host=el2003.bc
[bpm-t]
bpm-app1-t1 ansible_user=bpmadmin ansible_host=el2001.bc
bpm-app1-t2 ansible_user=bpmadmin ansible_host=el2003.bc
[bpm-all:children]
bpm-i
bpm-u
bpm-t
But ... running PB's becomes a mess now and cause errors. Logically I have two alias names to reach the same user/host combination : bpm-app1-u1 and bpm-app1-t1. I don't mind, that's perfectly logical, but if I were to test a new maintenance script, I would first push it to bpm-app1-i1 for testing and when ok, I probably would run the PB against bpm-all. But because of the non-unique user/host combinations for some aliases the PB would run multiple times on the same user/host. Depending on the actions in the PB this may work coincidentally, but it may also fail horribly.
Is there no way to tell Ansible "Run on ALL - UNIQUE user/host combinations" ?
Since most tasks change something on the remote host, you could use Conditionals to check for that change on the host before running.
For example, if your playbook has a task to run a script that creates a file on the remote host, you can add a when clause to "skip the task if file exists" and check for the existence of that file with a stat task before that one.
- Check whether script has run in previous instance by looking for file
stat: path=/path/to/something
register: something
- name: Run Script when file above does not exist
command: bash myscript.sh
when: not something.exists