We are using Terraform Enterprise Cloud and Azure DevOps YML pipelines for Azure infra deployments.
Requirement: We want to separate .tfvar files completely from the main terraform folder and keep them in different Repo called config Repository.
Solution 1: We can refer tfvars from the config repository while running the below command,
terraform plan --var-fil -We cannot implement
Note: Since we are using global templates, these terraform commands like fmt, validate, plan, and apply are managed by the template itself, we are not allowed to edit the template.
Here is the logic,
template expects only .tfvars file in the current directory, then there are some bash commands to rename it to .auto.tfvars.
We know that these auto.tfvars files will be automatically identified by Terraform.
Solution 2: We are expecting and struggling to implement and need some help
By default Template copies all terraform folders to ADO Agent Container. we want to make sure the .tfvar file from the Config repository is available in the agent container. Then this solution will be good.
May be,
We can achieve it by Copying the .tfvars file from the config repository to the agent container by writing some shell script. but it has to be inside the terraform folder. because only terraform folder will be copied to the agent container.
Or is there any way that we can integrate a shell script to terraform configuration which can download tfvars file from config repository to container in run time.
Any other solution or approach will be appreciated.
To make sure the config repo files are available during runtime you can add a second artifact to the release pipeline. This will allow you to modify your var argument with the appropriate file.
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/artifacts?view=azure-devops
One approach is to have your tfvars file stored as a secure file, then just add a step in your pipeline to download it, however, if you're using Terraform Enterprise, is there any particular reason to not use Terraform workspace variables?
Related
I have .terraform/modules folder generated by terraform itself.
It's where terraform keeps modules by default and I'm fine with that.
when running terraform init command and if .terraform folder is gone it will try to pull modules again I would like to avoid that step by saying to use pre-populated modules folder from different location - it's like building shared cache folder for terraform for our CI/CD pipelines, pull only if new version of a modules specified otherwise use cache.
NOTE:
We don't run anything on Jenkins locally, every `Stage` in Jenkins uses Ephemeral Docker
container agents to run all the `Steps` and to keep Jenkins clean,
otherwise I would use local workspace cache for all that.
is there a way to do that?
Thank you
How does the config file provider plugin work?
I have a Jenkinsfile for a declarative pipeline (multi-branch build) that contains:
configFileProvider([configFile(fileId: 'maven-settings', variable: 'MAVEN_SETTINGS')]) {
sh 'mvn -B -s $MAVEN_SETTINGS -DWHERE="$WHERE" deploy'
}
I have tried running this on two different Jenkins installations one which is installed directly and runs as daemon and another running as a container (jenkinsci/blueocean).
When run on the direct the Jenkins Config File Provider plugin is able to provide the required settings.
provisioning config files...
copy managed file [Maven settings] to file:/var/lib/jenkins/workspace/redacted#tmp/config8989354118161621860tmp
When run under the jenkinsci/blueocean container it fails with:
provisioning config files...
not able to provide the file [ManagedFile: id=maven-settings, targetLocation=null, variable=MAVEN_SETTINGS], can't be resolved by any provider - maybe it got deleted by an administrator
I have created a managed maven-settings.xml file with id maven-settings for blue/ocean but it is not being picked up.
I've also tried copying it to ~/.m2/settings.xml
By contrast the working installation does not have any managed files (settings.xml or otherwise) and I am unable to locate any maven settings file in the workspace. I'm not sure what the #tmp directory is. It is deleted by the time a build finishes.
So my question is:
Where should I put the settings to make the configFileProvider pass them on for the jenkinsci/blueocean build job?
How does the config file provider plugin work?
I have no idea what its doing so its hard to debug. The source is here but Java, Maven & Jenkins are not my main area.
What differences are there when Jenkins itself run as a container?
This answer suggests the the config file provider is unnecessary.
There is a similar question which is unanswered but it relates to a maven plugin.
I have part of the answer. My maven-settings file wasn't being picked up because I was using the file name rather than the file Id which is different.
The remaining part of the question is how is the original Jenkins instance able to generate this file without it being listed as a managed file.
I set a GIT_STRATEGY variable to none in a .gitlab-ci.yml file. So a repository is not built in a deployment server automatically. So we avoid building the git repository with its all history etc. A git-archive command will be used instead.
This way only needed files will be copied from a GitLab server.
Now I am looking for the way to send the bash script to the runner.
As we know, the .gitlab-ci.yml file is actually being sent to the runner. Isn't it a way to send the bash script to the runner too? Then I would make the .gitlab-ci.yml file just to start the bash script and have a job done.
Of course, I could put the commands to the yml, but there are some limitations then because of a yml syntax. I could also make the runner scp the file but this would be too complex. Any ideas?
update
Let me try to approach what I am going to achieve. There are several bash script files. Each of them is a specific scenario of the deployment. For example, one pulls project files from GitLab, deletes files in a folder public and moves project files to it.
The other script file pulls project files from GitLab, moves the files to the folder and creates a backup.
The other script file pulls project files from GitLab, moves the files, creates backups and performs a database migration. Etc.
There are three hosts:
Local PC of an administrator.
Remote GitLab server
Remote deployment server that server websites. (It has Nginx or Apache)
One of the script files will be executed on the deployment server. There is a number of the script files so we must provide replacing the script file from time to time.
The question is how to bring the script file to the deployment server so the runner could run it?
I am in process of moving configuration parameters out of Java application. I discover that the best approach is to extend your classpath and use .properties files (leave ZooKeeper alone for another requirement).
So my WAR file no longer have any hosts/IPs/URLs, users/passwords.
DevOps distribute configs manually across test, stage, stable installations.
Now time for Jenkins to run tests. But they fail as there are no required .propeties files in classpath.
How can I load this config files to Jenkins and how to make in available in test classpath?
maven-surefire-plugin allow extending classpath and passing system-properties.
So only question how to get separate directory in Jenkins hosting server and load files to this directory and create alias/placeholder/envvar per build job to refer to this path in build config.
This job can be done with SSH access, but I think that this is "wrong way". I expect that this can be done via Jenkins UI (any manager can upload file in WEB browser).
UPDATE I have no requirements for distributed slave/master builds but it whould nice to have solution that migrate configuration files to slaves automatically...
In this way sshing to host or ftp/scp - bad thing.
I read most of Jenkins docs, ask at mail list and IRC. Yea - Jenkins community is silent. At docs I found link to Config File Provider Plugin, after that I visit http://builder.evil.com/jenkins/pluginManager/available page and look for config keyword.
There are a lot related plug-ins with various usefulness to my subject (most useless first):
https://wiki.jenkins-ci.org/display/JENKINS/Envfile+Plugin - This plugin enables you to set environment variables via a file.
https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin - Allows credentials to be bound to environment variables for use from miscellaneous build steps.
https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin - Allows you to run a script before each build that generates environment variables for it.
https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin - This plugin makes it possible to have an isolated environment for your jobs.
https://wiki.jenkins-ci.org/display/JENKINS/Copy+Data+To+Workspace+Plugin - Copies data to workspace directory for each project build.
https://wiki.jenkins-ci.org/display/JENKINS/Copy+To+Slave+Plugin - This plugin allows to copy a set of files, from a location somewhere on the master node, to jobs' workspaces. It also allows to copy files back from the workspaces of jobs located on a slave node to their workspaces on the master one.
https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin - Adds the ability to provide configuration files (i.e., settings.xml for maven, XML, groovy, custom files, etc.) loaded through the Jenkins UI which will be copied to the job's workspace.
Only last plug-in - Config File Provider Plugin allow editing configs via Jenkins WEB interface. And it have brother - Managed Script Plugin - for uploading/managing/editing custom scripts. No question now I use Config File Provider Plugin!
You should keep the configs required for the tests together with the rest of source code, so that after compilation, your unit tests can run.
After deploying the .war, the DevOps team should overwrite the in-war configs with whatever per-environment configs that they have.
I have a bash script that I execute from a Jenkins job, using "execute shell".
The script starts an EC2 instance and sets an host variable containing the host name of the new instance.
I would like to set the host name of the new instance (script variable: host ) to a Jenkins environment variable so that I can pass it down to a downstream job (possibly using the Build Flow plugin).
Any idea how to do so?
Thanks
I ended up using a file to propagate data between builds.
The first build creates a file containing the information I need to propagate (host name of the newly created EC2 instance).
The file looks like:
host.name=ec2.123.3345.amazon.com
I use the EnvInject plugin to read the file and "inject" the properties that are then available in the next build (I'm using the Build Flow Plugin to orchestrate builds).
There is a plugin that you can install for inject your variable: EnvInject Plugin
If I understood your problem, I think this simple plugin is what you need.