I've got a CI script setup something like this with 3 files :
# file Vars
.def-vars:
STAGING_SSH_DEST: mysite.com
PROJECT_ROOT: myRoot
# file gitlab-ci
variables:
extends: .def-vars
STAGING_SSH_DEST: myrealsite.com
PROJECT_ROOT: /myRealRoot
deploy-stage:
extends: .deploy
variables:
SSH_DESTINATION: $STAGING_SSH_DEST
# file deploy
.deploy:
variables:
SSH_DESTINATION: mysite.com
RSYNC_DESTINATION: $SSH_DESTINATION:$PROJECT_ROOT
I have my files and variables split up like this to increase the re-usability of the scripts.
The idea was that since I have multi site destinations, staging prod, I want to be able to pass the ssh destination to each and have the job figure out the rsync on its own. Problem is, the variable expansion is not working the way I'd think it would.
In the deploy script I added a print and got the following :
$ echo $SSH_DESTINATION # This is the variable name local to job
myrealsite.com # Yep! printed the passed in value
$ echo $RSYNC_DESTINATION # $SSH_DESTINATION:$PROJECT_ROOT
$STAGING_SSH_DEST:/myRealRoot # That is the name of the variable passed in
The root and SSH_DESTINATION print just fine. When appending the two, the former seems to be expanded one too few times.
I've had the idea to just create the rsync variable within the script section but I'd like to avoid this as I want to be able to override the rsync variable without editing the .deploy job.
How can this be accomplished?
There is an issue with Gitlab CI variables, preventing you to expand variables correctly with your extends setup.
Your options to solve this are:
Use this solution with the help of before_script, posted inside aforementioned issue. Some limitations are there, but for simple stuff it is working pretty good.
before_script:
- export VAR1="${CI_PIPELINE_ID}"
- export VAR2="test-${VAR1}"
Do some kind of preparation via .env artifacts and downstream pipelines. This one is tougher to setup, but it will allow to create dynamic tasks (and pipelines), for example spawning multi-stage deploy after successful build.
"Setting a variable within another variable" should be easier with GitLab 15.6 (November 2022)
Support for special characters in CI/CD variables
Previously, it was difficult to use the $ character in a CI/CD variable because $ normally signifies the start of another variable.
GitLab would interpret it as a variable and try to expand it.
In this release, we are introducing the variable: expand: keyword which will allow you to mark a variable as “raw”.
A raw variable can contain any special characters and is not expanded when passed to the GitLab runner.
See Documentation and Issue.
And:
Example of variables:expand:
variables:
VAR1: value1
VAR2: value2 $VAR1
VAR3:
value: value3 $VAR1
expand: false
The result of VAR2 is value2 value1.
The result of VAR3 is value3 $VAR1.
Related
I have a variable group e.g. 'vargroup' with 3 variables (env ,envid, envpwd) in azure devops. i have declared the variable group in my yaml definiton too. If i use inline bash script these variables are being recognized and works fine.
However when I use the same script but located in the repository, the script does not recognize the variables in variable group.
variables:
- group: vargroup
- task: Bash#3
inputs:
filePath: 'script.sh'
my script calls these variables as $(env) , $(envid) , $(envpwd) .
what's the correct way to use the variables from variable group in a script path method?
After multiple tries , finally I have found the solution.
trick: using capital letters for passing the variables in the script.
my variables from variable group are env , envid & envpwd.
I have passed them as $ENV , $ENVID , $ENVPWD in my shell script and it worked.
Note: further to make a secret variable like $ENVPWD to work in azure yaml pipeline you will need to pass it as environment variable
Maybe you can change your script to take arguments and then pass your variables into your script:
script.sh $(env) $(envid) $(envpwd)
This is very funny but very frustrating problem. I am using an ENV variable, which specifies date. I provide an ISO 8601 compliant version and in application, I retrieve it and parse. When I specify it in GH action workflow, it is get parsed as a date (rather than a string) and formatted. Therefore, my application parsing fails.
Example:
.github/workflows/rust.yaml
env:
MY_DATE: '2020-10-07T12:00:00+01:00'
run: echo $MY_DATE
Result (GH action UI):
env:
TMOU_GAME_END: 10/07/2020 11:00:00
10/07/2020 11:00:00
It is specific to GitHub action and their yaml parsing, it works OK on Heroku, on various local setups, etc.
Things I tried and they don't work:
using no quotes, single quotes ('), double quotes (")
setting another ENV var, LC_TIME to en_DK.UTF-8
using !!str shorthand (see https://yaml.org/spec/1.2/spec.html, section Example 2.23. Various Explicit Tags); this one fails either with The workflow is not valid. .github/workflows/rust.yml: Unexpected tag 'tag:yaml.org,2002:str' or with The workflow is not valid. .github/workflows/rust.yml: The scalar style 'DoubleQuoted | SingleQuoted' on line 29 and column 24 is not valid with the tag 'tag:yaml.org,2002:str'
Is there any help? Any secret parameter I can turn on? Any escaping sequence? I just wanna GH Actions yaml parser to treat the value as a string.
Surprisingly, it seems GitHub Actions workflow YAML parser does not fully implement the standard and using explicit typing (like !!str) does not work. You can, however, workaround it by setting the environment variable to the desired value not in the YAML file itself but dynamically during workflow execution using a dedicated workflow command:
steps:
- name: Dynamically set MY_DATE environment variable
run: echo "MY_DATE=2020-10-07T12:00:00+01:00" >> $GITHUB_ENV
- name: Test MY_DATE variable
run: echo ${{ env.MY_DATE }}
This should do the trick.
I have Terraform that is using a templated bash script to set my user data section for an AWS launch configuration.
data "template_file" "user_data" {
template = "${file("${path.module}/user-data.tpl")}"
vars {
file_system_id = "${aws_efs_mount_target.my_efs_alpha.dns_name}"
}
}
The file_system_id variable then needs to be used in my template:
sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $$$${file_system_id}:/ /mnt/efs
Bash will interpret a single dollar sign as a bash variable. As I understand it, Terraform will interpret a double-dollar-sign as a Terraform variable. For added fun, the dollar signs in the template need to be escaped with another dollar sign -- hence the 4 dollar signs in front of file_system_id.
Looking at the user data in my Launch Config over in AWS Console, Terraform does not appear to be making any effort to replace my $$$${file_system_id) with the variable value from my template_file definition. Rather, it just shows up in the user data section as literally $${file_system_id}.
So, the question is, how do I get my EFS DNS name (or whatever other value I want) to replace the file_system_id variable in my template? What have I missed?
As BMW mentioned, you don't need to escape the dollar signs. ${file_system_id} works just fine.
Terraform's variable-replacement in templates will run first so you don't need to worry about how Bash will parse it until after the variables are replaced.
Is it possible to concatenate a string to call a Bamboo variable.
Using a script task in Bamboo, I want to generalize the following:
python my.py moon ${bamboo.mynamespace.moon}
to
SET planet=MOON
python my.py %planet% ${bamboo.mynamespace.%planet%}
But doing it like the second example above results in my python script receiving
${bamboo.mynamespace.%planet%}
as a string and not the value of
${bamboo.mynamespace.moon}
I know... moon is not a planet
I don't think it's going to be possible in the way how you're using it. Because once you use ${bamboo.variableName} Bamboo tries to resolve the variable and substitute it with a variable value. Since there's no variable%planet% Bamboo can't reference it.
But I think you could reorganise your solution a bit and make use environment variables (all Bamboo variables are passed to process as environment variables). So e.g. if Bamboo variable's name is variable.name you're allowed to reference to it via ${bamboo_variable_name} (bamboo prefix + all dots are replaced with underscore)
Then I can imagine you could get variable which interests you via print os.environ['bamboo_mynamespace_' + 'planet'] (more info on env variables in python here)
This is a system Octopus Deploy Variable:
#{Octopus.Action[Deploy To Server].Output.Package.InstallationDirectoryPath}
The text "Deploy to Server" is the name of the step in my project that deploys the Nuget Package to the server. This variable gives the install location of the NugetPackage.
I am wondering if I can make this more generic:
#{Octopus.Action[#{DeploymentStep}].Output.Package.InstallationDirectoryPath}
#{DeploymentStep} is itself a variable with the value of "Deploy to Server"?
I tried this and it not did do the substitution when it tried to run. But I am hoping there is a different syntax for variable in variable substitution.
(I want to do this so I can make this the default value for a Step Template.)
It can be done; but you need to use slightly different syntax!
Variable substitution syntax: http://docs.octopusdeploy.com/display/OD/Variable+Substitution+Syntax
$deploymentStep = "#{DeploymentStep}"
$installationDirectory = $OctopusParameters["Octopus.Action[$deploymentStep].Output.Package.InstallationDirectoryPath"]
Have just had the same issue, but with a few tests I got it working in a 1 liner.
You need to encapsulate the inner variable in brackets with a dollar, and you need to change the double quotes within the variables to single quotes so it doesn't complain about the miss match of quotes. Double quotes on the outside and single quotes in the in.
The example below gets a step name with an octopus variable and also a the machine name it ran on variable to produce the result:
$OctopusParameters["Octopus.Action[$($OctopusParameters['Octopus.Step.Name'])].Output[$($OctopusParameters['Octopus.Machine.Name'])].MyVarFromMachineFromStep"]