Jenkins Templates - maven

I am new to the Cloudbees Enterprise of Jenkins and to the concept of "templates".
I am trying to define a new template and this template will be used by 20-30 number of jobs. The job is a basic build job. After the build, I would like to have code analysis plugin. How can I define it in the Jenkins Template.
I can define it while creating a direct job in "Post Build Actions" but not sure how to define the same in a template.
Do you have any solutions/ suggestions ?

Cloudbees templates plugin is very powerful but not easy to master. Creation and administration of templates is not as user-friendly as one wishes it to be. ;)
For an introduction I recommend reading the following resources:
Basic concept
Template documentation
Tutorial for simple job template
Make sure you understand the difference between builder template and job template. I assume you want to create a number of jobs using the job template. Follow these steps:
First of all create a normal job that contains all actions you want to perform by all templated jobs.
Make sure this job works as expected for one example configuration.
Now create a new job template:
You will need to decide, which parts of the job configuration need to be adapted to each jobs configuration (e.g. source code repository). Create an parameter for each configuration option.
You might want to do some pre-processing on the job templates parameters using some transformation script - but we skip that for now.
Now you need to add an XML description of what the generated job should do. I recommend copying this XML description from our example job created in step #1. You can access it via this URL: http://your-jenkins/job/this-job/config.xml. Simply copy&paste the XML code in the browser. Newer Jenkins versions also allow to read the jobs XML configuration via the user interface.
Finally you need to fill in the templates arguments within the XML configuration. Simply replace the specific (hard-coded) values by a reference to the name of the templates parameters created before: ${param_name}
Save the template
Now create a new job. On the job creation screen you should be able to select your newly created job template as job type. After creating a job of the templates type you can define all template parameters for this specific job.
Try to run the template-based job and make sure it works as expected.
Create more template-based jobs as needed.
All template-based jobs share the build steps defined by the job template. If you change the job template later on, all depending jobs are updated accordingly. This is a very efficient way to administrate a large number of similar jobs. It is very much worth the effort. Good luck!

Related

I want to trigger multiple task runs that differ by their input parameters in a Databricks job

I am trying to run a job including a task that needs to multiple run in parallel using different parameter values.
I understand that this is possible based on this post:
https://docs.databricks.com/data-engineering/jobs/jobs.html#maximum-concurrent-runs
But I can't figure out how.
To create trigger on multiple jobs using Databricks UI, follow below path
Workflows > Jobs > Create
Here give Task name and select Type, Source and Path.
You can Add parameters as shown in the screenshot below.
In Advanced options you can Add dependent libraries, Edit email notifications, Edit retry policy, Edit timeout.

Create multiple MarkLogic Schedule Task for same module through ml-gradle

I am trying to create multiple instance of application on same marklogic environment. I can able to create all the configurations(users,roles,databases,forests,app servers...) but could not able to schedule individual tasks for separate database with same module path.
When tried to run ml-gradle mldeployApps failing at Tasks creation.
My whole application configuration will depends on from property file. for any APP-NAME a seperate insiance need to be created.
I tried deploying through ml-gradle
The mlDeployTasks is failing as already an task is available for the module path. When try to run secong with new failing as it is not recognizing task database
JSON:
{
"task-enabled":true,
"task-path":"/ext/schedules/monitor.xqy",
"task-root":"/",
"task-type":"daily",
"task-period":1,
"task-start-time": "10:00:00",
"task-database":"%%DATABASE%%",
"task-modules":"%%MODULES_DATABASE%%",
"task-user":"admin",
"task-priority":"normal"
}
ERROR:
Logging HTTP response body to assist with debugging: {"errorResponse":{"statusCode":"500", "status":"Internal Server Error", "messageCode":"MANAGE-INVALID", "message":"MANAGE-INVALID (err:FOER0000): task-database"}}
Error occurred while sending PUT request to /manage/v2/tasks/5389046897270663947/properties?group-id=Default; logging request body to assist with debugging: {
Expectation :
wants to deploy and undeploy whole application including schedules tasks based on APPLICATION-NAME as seperate instance
Actual:
the mlDeployTasks based on the module-path each task is identified with old existing database and fails to create a new task server.
Please suggest me the right way to achieve the same
MarkLogic's Management API is seeing your request as an attempt to change the task-database, but it only allows one property for a scheduled task to change (task-enabled). I think what you'll need to do here is have different task-path values for your different databases. That's not ideal, but if the implementation logic is all in a library that's imported by the task, the different modules themselves will be very lightweight.
Try ml-gradle 3.10.0 - support for this now exists - see the release notes for ml-app-deployer 3.10.0 (which provides most of the functionality in ml-gradle) - https://github.com/marklogic-community/ml-app-deployer/releases/tag/3.10.0

Where to put oozie.launcher.* configuration?

While trying to use Oozie properly, I ended up setting a few parameters, namely:
oozie.launcher.mapreduce.map.memory.mb
oozie.launcher.mapreduce.map.java.opts
oozie.launcher.yarn.app.mapreduce.am.resource.mb
oozie.launcher.mapred.job..queue.name
If I set them in the worfklow configuration, they work as expected.
Is there a way/a place to set them globally, ie. not per workflow? I was expecting that custom-oozie-site.xml would be the right place but apparently not (they have no effect if put there). Is the workflow itself the only place where they can be configured?
If it is relevant, I am using hdp 2.5.
In the Oozie Parameterization of Workflows section of the documentation, they state
Workflow applications may define default values for the workflow job parameters. They must be defined in a config-default.xml file bundled with the workflow application archive... Workflow job properties have precedence over the default values.
Another option I've seen done is defining a parent workflow definition and propagating to child workflows. Granted, this only works in specific instances and isn't always a good idea.
In addition the documentation notes in the Workflow Deployment section
The config-default.xml file defines, if any, default values for the workflow job parameters. This file must be in the Hadoop Configuration XML format. EL expressions are not supported and user.name property cannot be specified in this file. Any other resources like job.xml files referenced from a workflow action action node must be included under the corresponding path, relative paths always start from the root of the workflow application.
This is a problem my team is currently trying to fix across 12 different ETL loads.

Is there a way to import/export tasks from different CQ instances?

I have two instances of CQ and between them I want to be able to import/export tasks.
For example:
On instance 1 I can see all tasks by going to http://instance1/libs/cq/taskmanagement/content/taskmanager.html#/tasks/Delta
On instance 2 I can see all tasks by going to http://instance2/libs/cq/taskmanagement/content/taskmanager.html#/tasks/Delta
There might be some scenarios where I want to take all tasks from instance2 and add them as additional tasks into instance1 (on top of the tasks it may already have).
Is this possible to do?
Yes, you can do this with Package Manager. The tasks are stored as nodes in the JCR repository, so you can create a package that filters the task nodes you want to migrate from one instance to another. For example, you could define a package with this filter definition to include all tasks:
/etc/taskmanagement/tasks
If you don't want all tasks, you may need to define the filter(s) more narrowly to pick only the ones you want to include.
For example:
/etc/taskmanagement/tasks/2015-05-04/Delta/TheTaskYouWantToMigrate
Use the browser when defining the filter to find the tasks you want to include.
See Working with Packages for details on using the Package Manager. This Tutorial also shows how to create the package and add filters. Once you've created a package with the filters for the tasks you want to include, then build the package and download it. On your other instance upload the package you built and install it. You will then see the tasks one your first instance replicated onto the second instance.
Additionally to what Shawn said, you also can use replication mechanisms to do the work for you, and replicate the desired nodes between any two instances.

How to enqueue more than one build of the same configuration?

We are using two teamcity servers (one for builds and one for GUI tests). The gui tests are triggered from http GET in the last step of the build. (as in http://confluence.jetbrains.com/display/TCD65/Accessing+Server+by+HTTP)
The problem is, that there is only one configuration of the same kind in the queue at the same time. Is there a way to enable multiple starts of the same configuration? Can I use some workaround like sending a dummy id?
At the bottom the section "Triggering a Custom Build" here: http://confluence.jetbrains.com/display/TCD65/Accessing+Server+by+HTTP, you can find information about passing custom parameters to the build.
Just define some unused configuration parameter, like "BuildId", and pass, for example, current date (guid will work as well) to it
...&buildId=12/12/23 12:12:12

Resources