I've got Jenkins job A that triggers job B and afterwards executes a shell script.
Inside this shell script of Jenkins job A I want to use a variable set by Jenkins Job B.
How can I do this?
This can be accomplished in many ways. One way would be to configure Job A to have a build step, that triggers Job B, and fetches variables in a document after Job B has finished. Then Job A can read those variables and use them in later steps.
There are several things to consider here though. First of all this requires Job B to finish before Job A can/should continue, so if you are thinking of parallel job execution this isn't ideal. Secondly, when dealing with env variables you will need a plugin to make variables available outside of the build step (exporting isn't enough), check out the EnvInject plugin. And thirdly, if job configuration is becoming complex, there probably is a better way of doing it. With Jenkinsfile and previously pipelining plugins, Job orchestration has improved a lot, and passing parameters around and such is much easier in this new, shiny world. That being said, here is an example of something that works like what you are asking about.
Job A
As a build step, trigger Job B, and let Job A halt while Job B finishes
As the next build step, copy an artifact from another build (Job B latest stable), using the Copy Artifact Plugin.
Do something with the file, for example just printing it's content, it's now accessible in Job A.
Job B
Export a variable and save it to a file
Archive the written file and make it accessible to Job A.
This isn't pretty, but it works at least.
P.s. I'd recommend checking out the Jenkinsfile (https://jenkins.io/doc/book/pipeline/jenkinsfile/) options, it simplifies a lot after the initial learning curve.
Related
Since there are too many jobs on Azkaban, I have to test new jobs one by one manually.
Assume I upload some new jobs and is it possible to write a Python (or any other language) script to fetch the dependencies between these jobs and then run them on Azkaban in paralell?
For instance, there are there jobs a, b, c and b dependents on a. They are supposed to be scheduled like:
Starts to run job a and job c
When job a finishes, starts to run job b.
I did not find any helpful info or API on the Azkaban official website (Maybe I missed useful info).
Any help is appreciated.
A certain number of jobs needs to be executed in a sequence, such that result of one job is input to another. There's also a loop in one part of job chain. Currently, I'm running this sequency using wait for completition, but I'm going to start this sequence from web service, so I don't want to get stuck waiting for response. I wan't to start the sequence and return.
How can I do that, considering that job's depend on each other?
The typical approach I follow is to use Oozie work flow to chain the sequence of jobs with passing the dependent inputs to them accordingly.
I used a shell script to invoke the oozie job .
I am not sure about the loops within the oozie workflow. but the below link speaks about the way to implement loops within the workflow.Hope it might help you.
http://zapone.org/bernadette/2015/01/05/how-to-loop-in-oozie-using-sub-workflow/
Apart from this the JobControl class is also a good option if the jobs need to be in sequence and it requires less efforts to implement.It would be easy to do loop since it would be fully done with Java code.
http://gandhigeet.blogspot.com/2012/12/hadoop-mapreduce-chaining.html
https://cloudcelebrity.wordpress.com/2012/03/30/how-to-chain-multiple-mapreduce-jobs-in-hadoop/
I am aware of the BUILD_NUMBER environment variable; however, unless I'm mistaken, this is a global variable which appears to be shared among all projects. This is a problem since, if I have two agents running jobs, one script may try to use the current project's build number after another script has updated it for a different project.
Here is an example scenario:
Job J1 starts, executing some script S. BUILD_NUMBER is currently, let's say, 500.
Someone manually starts job J2, which now sets BUILD_NUMBER to something completely different, say, 750.
Script S, which is still running, now reaches the point where it actually uses BUILD_NUMBER, now set to 750 (an impossible value for job J1, which is only up to 500).
Am I correct in this understanding? If so, how do I address this problem?
BUILD_NUMBER is not global to the system. It's local to each job. In your example, J1 and J2 will have different BUILD_NUMBER (J1=500, J2-750) values and they will remain constant - within the job - for the duration of the job.
You can get the latest (completed or currently executing) build number with:
http://[JENKINS_URL]/job/[JOB_NAME]/lastBuild/buildNumber
or from within the job itself, you can use environment variable:
${BUILD_NUMBER}
You can use all other permalinks too, like:
/lastStableBuild/
/lastSuccessfulBuild/
/lastFailedBuild/
and so on.
Unless you are customizing your build number with a plugin such as Next Build Number, your next build number will always be an increment to the latest build number.
The build numbers are per job. They are not global environment variables.
Every time any job is executed, it copies/inherits existing environment variables into a new session. Jenkins/Hudson then adds all job parameters and extra variables (including ${BUILD_NUMBER}) to that new session. They are available as environment variables only for the duration of that session (i.e. for the duration of the job run). Once the job completes, this session is destroyed (along with all the extra variables that were generated for the session).
You cannot access the environment variables of one job-run/session from another job-run/session. You can however access the /lastBuild/buildNumber of another job through the URL method I provided. You can also access /lastBuild/buildTimestamp the same way.
To access job parameters of another job, you can use:
http://[JENKINS_URL]/job/[JOB_NAME]/lastBuild/api/json
or
http://[JENKINS_URL]/job/[JOB_NAME]/lastBuild/api/xml
but you would need to parse the output for what you need.
Use the Get Next Build Number plugin.
https://wiki.jenkins-ci.org/display/JENKINS/Next+Build+Number+Plugin
I'm using LoadLeveler to submit jobs on an IBM/BlueGene architecture. I read the documentation made from IBM and also gave Google a try, but I cannot find how to do the following, which I expect should be there:
One can use the
queue
keyword to tell LoadLeveler that a new job step is described, so I could do something like
first_step
queue
second_step
queue
but what I fail to find is a way that does something like
loop job_id = 1,10
do_job_with_given_job_id
end
Do I have to write a "normal" shell script that in turn calls a load level script for a bunch of times, or is there some built in loop mechanism? I know that other job managers can do this.
When this comes up, we normally just recommend that one writes a shell script which generates the job submission script or scripts; that's what I do for my own jobs. Do these steps have dependancies on each other?
Also, just out of curiosity, which schedulers/resource managers can queue multiple jobs within a loop in a submission script? Not the PBS-based ones...
I have separated a big Hudson job into smaller jobs. Job A does the main build and Job B does another build with a different configuration. I have configured Hudson, so that the A triggers B and that works fine, the problem is that Job A has the original build number and B has started from 1.
My question is: Is it possible to pass the BUILD_NUMBER environment variable somehow from Job A to Job B? The build number is used in the build artifact names, hence it would be nice to have the numbers match between artifacts.
Thanks.
Use the parametrized Parameterized Trigger Plugin, which will allow you to pass the build number from A to B. You will not be able to actually set the build number in job B, but you will have the build number from A to generate your version number.
If you want to synchronize the build number, you can edit the file nextBuildNumber in the job directory to match the number from job A. Be aware that these numbers will drift apart over the time since when A fails B will not be started.
EDIT I just stumbled across the Next Build Number Plugin. Have a look, If this one helps you.