I have .terraform/modules folder generated by terraform itself.
It's where terraform keeps modules by default and I'm fine with that.
when running terraform init command and if .terraform folder is gone it will try to pull modules again I would like to avoid that step by saying to use pre-populated modules folder from different location - it's like building shared cache folder for terraform for our CI/CD pipelines, pull only if new version of a modules specified otherwise use cache.
NOTE:
We don't run anything on Jenkins locally, every `Stage` in Jenkins uses Ephemeral Docker
container agents to run all the `Steps` and to keep Jenkins clean,
otherwise I would use local workspace cache for all that.
is there a way to do that?
Thank you
Related
We are using Terraform Enterprise Cloud and Azure DevOps YML pipelines for Azure infra deployments.
Requirement: We want to separate .tfvar files completely from the main terraform folder and keep them in different Repo called config Repository.
Solution 1: We can refer tfvars from the config repository while running the below command,
terraform plan --var-fil -We cannot implement
Note: Since we are using global templates, these terraform commands like fmt, validate, plan, and apply are managed by the template itself, we are not allowed to edit the template.
Here is the logic,
template expects only .tfvars file in the current directory, then there are some bash commands to rename it to .auto.tfvars.
We know that these auto.tfvars files will be automatically identified by Terraform.
Solution 2: We are expecting and struggling to implement and need some help
By default Template copies all terraform folders to ADO Agent Container. we want to make sure the .tfvar file from the Config repository is available in the agent container. Then this solution will be good.
May be,
We can achieve it by Copying the .tfvars file from the config repository to the agent container by writing some shell script. but it has to be inside the terraform folder. because only terraform folder will be copied to the agent container.
Or is there any way that we can integrate a shell script to terraform configuration which can download tfvars file from config repository to container in run time.
Any other solution or approach will be appreciated.
To make sure the config repo files are available during runtime you can add a second artifact to the release pipeline. This will allow you to modify your var argument with the appropriate file.
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/artifacts?view=azure-devops
One approach is to have your tfvars file stored as a secure file, then just add a step in your pipeline to download it, however, if you're using Terraform Enterprise, is there any particular reason to not use Terraform workspace variables?
I am trying to maintain my changes to config files of resource.bundle directories from remote cocoapods repositories.
While working on the implementation I am able to make changes locally but I do not own the external repository.I would like to be able to refer to the code owners tags implementing the pods from their repos in my project while maintaining my configuration changes.
It has been suggested to me to create a script phase in my build process that would copy files from a "assets folder" within the project to the finished pod directory after the remote pull and build.
This sounds feasible but I am not sure where to start in this process or what the script would like.
essentially I would have a
root/assetsfolder/resource.bundle
that would need to be copied to
Pods/ExternalPodName/Core/resource.bundle
Any help would be appreciated.
From the official document from GitLab, it seems that a CI/CD stage can only be skipped based on if a file is changed. Is it possible to skip a step based on if a file / folder exists on the machine to be deployed?
The case is that it is common to use a package management tool. (e.g. composer in PHP or NPM in Node.js) Currently the rule is to check if the respective configuration file is changed (e.g. composer.json for composer or package.json for NPM) to see if it is necessary to run the install step (i.e. composer install or npm install) However, although it seldom happens, when a new machine is used for deployment, the CI/CD would crash because the install step is skipped.
Currently, the problem is solved by manually trigger the install step, but is it possible to auto detect if the file exists at the hosting machine to determine if the install step should be run?
I don't think that there is a way to do what you describe. The only/except feature in GitLab runs on the GitLab server, before any jobs are created. To be able to know if a file already exists on a server, there needs to be a job created and it needs to be assigned to a gitlab runner.
They have this issue, where it looks like it will be possible to start a job and then determine that it should be skipped by using exit codes:
https://gitlab.com/gitlab-org/gitlab/issues/16733
In the mean time, you can make it so that your install job always runs, and then just use your build scripts to check for this file. From the pipeline view in GitLab, you will always see your install step. The actual job can just skip to do anything if the file already exists, and then you don't need to remember to run the job manually.
I have configured and working following setup
gitlab-ci, which uses docker-machine runner and uploads cache to S3
maven build with configured caching
caching correctly loads and uploads on each job
But the problem is, that every time I run mvn install, something in the local maven repository changes (I assume it updates pom metadata) and gitlab runner keeps uploading new versions of the cache, on every single build.
It is still faster and more reliable to use this "busted" cache, than to download the deps from internet every time, but the upload can take a long time and I would like to shave off this extra time.
How can I modify my build to force maven, to generate cacheable local repository?
Simplified version of my .gitlab-ci.yml:
variables:
# we have a custom java+maven image, that uses this ENV variable,
# to auto-configure path where to put the local maven repository
MAVEN_LOCAL_REPOSITORY: $CI_PROJECT_DIR/.cache/maven
job-build:
stage: build
image: internal-gitlab/java/maven:3.6-jdk8-alpine
script:
- mvn -B clean package
cache:
key: backend-dependencies
paths:
- .cache/
You have a constant as a cache key. Maybe a more fine grained cache would help.
See the link here
In short - prepare your own maven image with required dependencies and use it instead of internal-gitlab/java/maven:3.6-jdk8-alpine.
Some details:
First of all, you need to create a maven docker image where all (or most of) required for your project dependencies are presented. Publish it to your registry (gitlab has one) and use it instead of internal-gitlab/java/maven:3.6-jdk8-alpine.
To create such an image I usually create an additional job in CI triggered manually. You need to trigger it at initial stage and when project dependencies are heavily modified.
Working sample can be found here:
https://gitlab.com/alexej.vlasov/syncer/blob/master/.gitlab-ci.yml
- this project is using the prepared image and also it has a job to prepare this image.
https://gitlab.com/alexej.vlasov/maven/blob/master/Dockerfile
- dockerfile to run maven and download dependencies once.
The pros:
don't need to download dependencies each time - they are inside a
docker image (and docker layers are cached on the runners)
don't need to upload artifacts when job is finished
I am in process of moving configuration parameters out of Java application. I discover that the best approach is to extend your classpath and use .properties files (leave ZooKeeper alone for another requirement).
So my WAR file no longer have any hosts/IPs/URLs, users/passwords.
DevOps distribute configs manually across test, stage, stable installations.
Now time for Jenkins to run tests. But they fail as there are no required .propeties files in classpath.
How can I load this config files to Jenkins and how to make in available in test classpath?
maven-surefire-plugin allow extending classpath and passing system-properties.
So only question how to get separate directory in Jenkins hosting server and load files to this directory and create alias/placeholder/envvar per build job to refer to this path in build config.
This job can be done with SSH access, but I think that this is "wrong way". I expect that this can be done via Jenkins UI (any manager can upload file in WEB browser).
UPDATE I have no requirements for distributed slave/master builds but it whould nice to have solution that migrate configuration files to slaves automatically...
In this way sshing to host or ftp/scp - bad thing.
I read most of Jenkins docs, ask at mail list and IRC. Yea - Jenkins community is silent. At docs I found link to Config File Provider Plugin, after that I visit http://builder.evil.com/jenkins/pluginManager/available page and look for config keyword.
There are a lot related plug-ins with various usefulness to my subject (most useless first):
https://wiki.jenkins-ci.org/display/JENKINS/Envfile+Plugin - This plugin enables you to set environment variables via a file.
https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin - Allows credentials to be bound to environment variables for use from miscellaneous build steps.
https://wiki.jenkins-ci.org/display/JENKINS/Environment+Script+Plugin - Allows you to run a script before each build that generates environment variables for it.
https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin - This plugin makes it possible to have an isolated environment for your jobs.
https://wiki.jenkins-ci.org/display/JENKINS/Copy+Data+To+Workspace+Plugin - Copies data to workspace directory for each project build.
https://wiki.jenkins-ci.org/display/JENKINS/Copy+To+Slave+Plugin - This plugin allows to copy a set of files, from a location somewhere on the master node, to jobs' workspaces. It also allows to copy files back from the workspaces of jobs located on a slave node to their workspaces on the master one.
https://wiki.jenkins-ci.org/display/JENKINS/Config+File+Provider+Plugin - Adds the ability to provide configuration files (i.e., settings.xml for maven, XML, groovy, custom files, etc.) loaded through the Jenkins UI which will be copied to the job's workspace.
Only last plug-in - Config File Provider Plugin allow editing configs via Jenkins WEB interface. And it have brother - Managed Script Plugin - for uploading/managing/editing custom scripts. No question now I use Config File Provider Plugin!
You should keep the configs required for the tests together with the rest of source code, so that after compilation, your unit tests can run.
After deploying the .war, the DevOps team should overwrite the in-war configs with whatever per-environment configs that they have.