We are having issues setting the Jenkins environment variables on our dynamic EC2-Fleet.
We already have a fixed master (linux) and a fixed Windows slave but wanted to add slaves dynamically when the load on the system becomes heavy.
For this we created a Spot Request Instance in AWS spinning up linux machines from an AMI and control this via the EC2-fleet-plugin in Jenkins.
Before this EC2-fleet can be of any help, our jobs must be able to run on its nodes.
Most of our jobs use Jenkinsfiles and need certain environment variables to be set but the EC2-fleet-plugin does not provide the possibility to set environment variables (https://issues.jenkins-ci.org/browse/JENKINS-36544).
As suggested on this ticket (JENKINS-36544), we tried to set the environment variables in "System Configuration" for the dynamic ec2 slaves and set the environment variables for the other nodes on the "Node Configuration" overriding the "System Configuration", or so we thought.
This should work if this bug wouldn't exist: https://issues.jenkins-ci.org/browse/JENKINS-44425. Because of this bug the "System Configuration" overrides the "Node configuration" instead of vice versa. So we can't use this as the existing nodes would not have the correct environment variables anymore.
As a last resort we tried to set the environment variables on the dynamic ec2 slaves by creating an /etc/profile.d/jenkinsvars.sh on the AMI used by the Spot Request Instance.
This script would be automatically run on login system wide (https://help.ubuntu.com/community/EnvironmentVariables#A.2Fetc.2Fprofile.d.2F.2A.sh).
Next to that we also attempted to set them in /home/ubuntu/.profile on the AMI singling out the ubuntu user which is the user running the Jenkins agent (https://help.ubuntu.com/community/EnvironmentVariables#A.2BAH4-.2F.profile).
But it appears Jenkins does not use these environment variables but its own...
A way that works is to adapt the jobs to load a groovy file that's part of the AMI to set the environment variables we need but that would mean to change almost all jobs we have, next to all Jenkinsfiles that are included in our repositories (Bitbucket project).
We would like to avoid this....
Try the following strategy:
Leverage User Data to run a shell script when the Spot Instance launches. It is the primary recommended way by Amazon and the plugin authors.
Instead of saving variables into the environment, have the user data script save them into either /var/tmp or /etc/profile or in parameter store. Refer to answers in this SO question. If you want to encrypt your info use KMS parameter store, if you dont care use one of the others.Choose one of the answers to best fit your needs.
Alter your Jenkins job to pause until your user data script has completed running (refer to the documentation from the plugin)
Change your Jenkins job to pick up the variables from the location you chose in step 2.
Try restarting the server environment.
Just saying.
So we can't use this as the existing nodes would not have the correct environment variables anymore.
Update your existing nodes to load the environment variables when they are provisioned / started, then remove them from the System configuration, then add them to the Node configuration.
You could also try setting the Slave command prefix field to ENV_VAR1=val1 ENV_VAR2=val2, although I haven't tried that.
Thirdly, you can try putting your variables directly into /etc/profile which should always be loaded no matter what user you are logging in as.
However, the easiest by far is to make all of your drones/agents exactly the same and set your environment variables in whatever scripts you run to build your projects. Use docker to pull dependencies to the agents as necessary during the job and to set up specific environments for your applications. This greatly simplifies the maintenance and configuration of your agents.
The Jenkins version or version of the EC2 plugin are missing in the question, but according to the description in this merged pull request, this bug should be fixed now: https://github.com/jenkinsci/ec2-plugin/pull/440#issuecomment-597160730
Jenkins version:
so this change works in both <=2.204, and >=2.205
EC2 plugin version: >=ec2-1.50
JENKINS-36544 - Fix Node Properties on Jenkins 2.205+ (#440) #jhansche
From the Pull Request description:
Navigate to the cloud configuration screen (this moves to a new page >=2.205)
Click "Add a new cloud"
Click "Amazon EC2"
Under the "AMIs" section, click "Add"
At the bottom of the AMI block, expand "Advanced"
Expect to see "Node Properties" block at the bottom of the block
Node Properties has the Environment variables.
Related
On Jenkins it is possible to set same arguments per agent but with different values e.g
Mac agent:
JENKINS_REPOS=/Users/johndoe/Repositories
UNITY_VERSIONS_PATH=/Users/johndoe/Editors
Windows agent:
JENKINS_REPOS=C:\Program Files\android\Repositories
UNITY_VERSIONS_PATH=C:\Program Files\android\Editors
once done, I can run the same pipeline on both agents and values change dynamically, according to agent I specify before build starts e.g agent Mac
I saw there is an option to define arguments on agents, however it does not really work except for system.agent.checkoutDir - I can set custom directory for each build.
How can I do it in same way on Teamcity? Thanks in advance.
I search a lot on the web, almost all links says define JVM custom variables in jvm.options also placed it on ${server.config.dir}/jvm.options.For example I added a variable called -DAPP_ENV=PROD. But this is getting as NULL after server startup.
Any idea?
It looks like you want to define an environment variable, so you have two options.
1. Use an Environment variable
In this case, you can define an environment variable (like $PATH) and load it in your app. Note this is not a JVM argument, and it will be set in the bin/server shell command used to start the server.
In the file:${server.config.dir}/server.env
Add the following line: APP_ENV=PROD
Access the value with:
System.getenv("APP_ENV"); -> PROD
2. Use a System property
This is what you are trying to do, so I am not sure why it doesn't work for you, but here's how:
In the file:${server.config.dir}/jvm.options
Add the following line: -DAPP_ENV=PROD
Access the value with:
System.getProperty("APP_ENV"); -> PROD
Note that in both cases these values are set at server start-up, and they are not changed dynamically (most Liberty configuration is dynamic). The JVM options and environment are sourced and set during the start script so a restart is required if you want to change either one.
My personal recommendation is go to the server.env route - its more generic and (to me) feels more appropriate since you are trying to influence the execution environment of the process, rather than defining behaviors or configuration of the JVM.
I'm using talend enterprise edition, I'm trying to use system environment variables as parameters for my jobs. when using system.getenv("paramname") and runnig the job, I'm getting the values from my local machine. what I need to do to to get the values from the talend server machine. The idea is to centrally add all the parameters as environment variables in the talend server and all users should use those env variables as parameters. any input is appreciated.
Instead of system.getenv("paramname"), please use system.getProperty("paramname"). As system.getenv() is deprecated
Hope this helps...
This thread might also be of use as it could be used to accomplish similgar goals Reading properties from an external file . I included screenshots and description in that answer.
This is a similar approach, but allows placing a common.properties (or other named file) for all Talend jobs running on that same job server to use. This also makes it easy to have different Talend job servers (dev, qa, production, etc), where the same jobs are installed, but they pull their correct settings from the common property file (environment dependent).
It uses the components tFileInputDelimited and tContextLoad to accomplish the task.
We're using TeamCity 9 and have a pool of agents, only some of which can run a particular job we've created. It's going to be a while before a new pool can be created with new agents, and in the meantime we would like to set this job to run only on specific ones.
In the configuration settings we can choose system.agent.name, but that only allows us one agent. Is there a way to configure things so that I can run on multiple agents based on their name, so that, say teamcity_agent01, teamcity_agent07 and teamcity_agent12 are eligible?
If the capabilities you want on the agent expose environment variables you can have a command line step refer to the environment variable and that will scope the job to a suitable agent.
e.g.
echo "%env.PATH_TO_GIT%"
You can add environment variables to your agents like this
open file “/conf/buildAgent.properties” at agent
system. This file contains parameters in key=value format.
To add a system properties add line like -> system.os=Linux
To add an environment variable add line like ->
env.JAVA_HOME=/home/java/jdk1.6.0_13 the
The beauty of this is that as you add more capabilities and agents then you don't need to revisit each job to change where it can run
For Teamcity 9.x
Next to the Projects, and Changes select Agents.
Select the agent that you want to run specific jobs on.
Select the "Compatible Configurations" tab.
On the "Current run configuration policy" dropdown select "Run assgined configurations only"
Click on the "+ Assign configuration" button.
Compatible configurations are colored blue. Select as you wish.
The timing of the configurations and when they should run can be configured under each build configuration triggers tab.
I have a Test::Unit configuration and at runtime, I would like a prompt in which i enter an environment(e.g. platform, staging or production) and the test runs in the specified environment. The related links for all environments are in environments.yml in the codebase
Right now, if I manually add an environment variable called DOMAIN and assign its value to 'platform', the test runs in the specified environment.
To do what I need to do, I have tried the following till now:
Created a Shell script which sets the DOMAIN env variable
In Before launch in IDEA, I set it to an external tool.
That tool calls the script created in step 1. A prompt is created and this takes in the environment to run the test. That env is passed as a parameter to the shell script.
The problem is, Idea does not pick up the change. Is there an easy way of doing this?