Bash - Gitlab CI not converting variable to a string - bash

I am using GitLab to deploy a project and have some environmental variables setup in the GitLab console which I use in my GitLab deployment script below:
- export S3_BUCKET="$(eval \$S3_BUCKET_${CI_COMMIT_REF_NAME^^})"
- aws s3 rm s3://$S3_BUCKET --recursive
My environmental variables are declared like so:
Key: s3_bucket_development
Value: https://dev.my-bucket.com
Key: s3_bucket_production
Value: https://prod.my-bucket.com
The plan is that it grabs the bucket URL from the environmental variables depending on which branch is trying to deploy (CI_COMMIT_REF_NAME).
The problem is that the S3_BUCKET variable does not seem to get set properly and I get the following error:
> export S3_BUCKET=$(eval \$S3_BUCKET_${CI_COMMIT_REF_NAME^^})
> /scripts-30283952-2040310190/step_script: line 150: https://dev.my-bucket.com: No such file or directory
It looks like it picks up the environmental variable value fine but does not set it properly - any ideas why?

It seems like you are trying to get the value of the variables S3_BUCKET_DEVELOPMENT and S3_BUCKET_PRODUCTION based on the value of CI_COMMIT_REF_NAME, you can do this by using parameter indirection:
$ a=b
$ b=c
$echo "${!a}" # c
and in your case, you would need a temporary variable as well, something like this might work:
- s3_bucket_variable=S3_BUCKET_${CI_COMMIT_REF_NAME^^}
- s3_bucket=${!s3_bucket_variable}
- aws s3 rm "s3://$s3_bucket" --recursive

You are basically telling bash to execute command, named https://dev.my-bucket.com, which obviously doesn't exist.
Since you want to assign output of command when using VAR=$(command) you should probably use echo
export S3_BUCKET=$(eval echo \$S3_BUCKET_${CI_COMMIT_REF_NAME^^})
Simple test:
VAR=HELL; OUTPUT="$(eval echo "\$S${VAR^^}")"; echo $OUTPUT
/bin/bash
It dynamically creates SHELL variable, and then successfully prints it

Related

Pass file variable to gitlab job

I am having trouble with dynamically passing one of two file based variables to a job.
I have defined two file variables in my CI/CD settings that contain my helm values for deployments to developement and production clusters. They are typical yaml syntax, their content does not really matter.
baz:
foo: bar
I have also defined two jobs for the deployment that depend on a general deployment template .deploy.
.deploy:
variables:
DEPLOYMENT_NAME: ""
HELM_CHART_NAME: ""
HELM_VALUES: ""
before_script:
- kubectl ...
script:
- helm upgrade $DEPLOYMENT_NAME charts/$HELM_CHART_NAME
--install
--atomic
--debug
-f $HELM_VALUES
The specialization happens in two jobs, one for dev and one for prod.
deploy:dev:
extends: .deploy
variables:
DEPLOYMENT_NAME: my-deployment
HELM_CHART_NAME: my-dev-chart
HELM_VALUES: $DEV_HELM_VALUES # from CI/CD variables
deploy:prod:
extends: .deploy
variables:
DEPLOYMENT_NAME: my-deployment
HELM_CHART_NAME: my-prod-chart
HELM_VALUES: $PROD_HELM_VALUES # from CI/CD variables
The command that fails is the one in the script tag of .deploy. If I pass in the $DEV_HELM_VALUES or $PROD_HELM_VALUES, the deployment is triggered. However if I put in the $HELM_VALUES as described above, the command fails (Error: "helm upgrade" requires 2 arguments, which is very misleading).
The problem is that the $HELM_VALUES that are accessed in the command are already the resolved content of the file, whereas passing the $DEV_HELM_VALUES or the $PROD_HELM_VALUES directly works with the -f syntax.
This can be seen using echo in the job's output:
echo "$DEV_HELM_VALUES"
/builds/my-company/my-deployment.tmp/DEV_HELM_VALUES
echo "$HELM_VALUES"
baz:
foo: bar
How can I make sure the $HELM_VALUES only point to one of the files, and do not contain the files' content?

GitHub Actions to use variables set from shell

Goal:
In GitHub Actions, to define my commit message dynamically from shell:
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on $(date -I)"
However, it seems that I have to define a environment variable then use it. I'm following How do I set an env var with a bash expression in GitHub Actions? and other help files like this, but still cannot tell how to make use of such environment variable that I've define previously. This is what I tried but failed:
- name: Checkout repo
uses: actions/checkout#v2
- run: |
touch sample.js
echo "today=$(date -I)" >> $GITHUB_ENV
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on ${today}"
How to make it works?
If you want to reference an environment variable set using the $GITHUB_ENV environment file in the arguments to another task, you'll need to use workflow syntax to access the appropriate key of the top level env key, like this:
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on ${{env.today}}"
You can access it as a standard environment from inside of a running task, for example:
- name: Show an environment variable
run: |
echo "today is $today"
In that example, the expression $today is expanded by the shell,
which looks up the environment variable named today. You could also
write:
- name: Show an environment variable
run: |
echo "today is ${{env.today}}"
In this case, the expansion would be performed by github's workflow
engine before the run commands execute, so the shell would see a
literal command that looks like echo "today is 2021-07-14".
You can accomplish something similar using output parameters, like this:
- name: "Set an output parameter"
id: set_today
run: |
echo "::set-output name=today::$(date -I)"
- name: Commit changes
uses: EndBug/add-and-commit#v7
with:
message: "added on ${{steps.set_today.outputs.today}}"
Using output parameters is a little more granular (because they are
qualified by the step id), and they won't show up in the environment
of processes started by your tasks.

Chef - use env variable with package resource what created with bash resource

I would get the changed project from git commit and install the package based on this.
Here is my code
bash 'get_project' do
code <<-EOH
filelist=$(git diff-tree --no-commit-id --name-only -r $1)
for file in ${filelist[#]}; do
project=$(echo $file | cut -d "/" -f1)
projectList+=($project)
done
for changedProject in $(echo "${projectList[#]}" | sort | uniq); do
INSTALLABLE_RPM=application-$changedProject
done
EOH
environment 'INSTALLABLE_RPM' => '$INSTALLABLE_RPM'
end
zypper_package ENV['INSTALLABLE_RPM']
My idea is to generate the INSTALLABLE_RPM variable with bash and install the package with zypper. Unfortunately it doesn't work. The zypper_package resource cant recognize.
I ran out of ideas :-(
The environment property of the bash resource is to supply existing environment variables to execute the bash command(s).
(These variables must exist for a command to be run successfully.)
Specifying environment variables here will not set them in the shell. Also from within the code block you will not be able to access the Ruby's ENV hash.
There may not be a straight-forward way to do this. One of the options is to write this package (list?) to a file. Then we can read the file contents into variable, and use it with zypper_package resource.
Example:
Since you have used for loop in Shell, I believe you get a list of packages, so I am considering pkg_list as Array. I've set compile_time to true as the variable assignment below bash resource will only run during compile time.
bash 'get_project' do
code <<-EOH
# your code as-it-is
for changedProject in $(echo "${projectList[#]}" | sort | uniq); do
echo "application-$changedProject" >> /tmp/rpm_packages
done
EOH
compile_time true
end
pkg_list = File.read('/tmp/rpm_packages').split
zypper_package pkg_list
# remove the file for good measure :)
file '/tmp/rpm_packages' do
action :delete
end

Terraform to read variables from environment

I have written a terraform configuration with variable definition like:
variable "GOOGLE_CLOUD_REGION" {
type = string
}
When I run terraform plan I am asked to fill in this variable even though this variable is set within my environment.
Is there a way to tell terraform to work with current env vars? Or do I have to export them and pass them somehow manually one-by-one?
You can define the environment variable TF_VAR_GOOGLE_CLOUD_REGION to set that variable.
If you are using bash, it might look like this:
export TF_VAR_GOOGLE_CLOUD_REGION="$GOOGLE_CLOUD_REGION"
terraform apply ...
From Environment Variables under Configuration Language: Input Variables.
As a fallback for the other ways of defining variables, Terraform searches the environment of its own process for environment variables named TF_VAR_ followed by the name of a declared variable.
This can be useful when running Terraform in automation, or when running a sequence of Terraform commands in succession with the same variables. For example, at a bash prompt on a Unix system:
$ export TF_VAR_image_id=ami-abc123
$ terraform plan
...
You can create a file that ends with .tfvars or .tfvars.json and then when you run a plan you specify that file:
terraform apply -var-file="example.tfvars"
If you name the file terraform.tfvars or terraform.tfvars.json or have a file with names ending in .auto.tfvars or .auto.tfvars.json
then Terraform automatically loads the variable definition file and you don't have to manually specify it when you run a plan.
An example of what the terraform.tfvars file will look like:
first_env_var = "environment_variable_one"
second_env_var = "environment_variable_two"
An example of what the terraform.tfvars.json file will look like:
{
"image_id": "ami-abc123",
"availability_zone_names": ["us-west-1a", "us-west-1c"]
}
I would approach this by creating a variables.tf file, within the project directory. with the required variable block you can specify a default:
variable "GOOGLE_CLOUD_REGION" {
type = string
default = "us-west1"
}
this will then be used as the default value during each run, and you will not be prompted.

bash use ruby code and save as variable

my end goal is to parse a visual studio team services ssh git url and use it to clone origin and my fork. I'm in windows and I use git bash I've made a few shell scripts to help me to clone it. Before when we used gitweb it was easy for me to parse as I could either run git_clone myproject or git_clone myproject.git or git_clone git://ourgitserver.ourcompany.com/myproject.git and the script would clone the above as origin and also add a remote with my user name in the form of ssh://git#outgitserver.ourcompany.com/myproject.git (and it handled name spaces well too). Well we started using vsts and I want to do the same thing.
The git_clone method changed a few times because of how people would tell/im/email me the link for the git project. I wanted to be able to just copy and paste it with minimal changes. thus far I have a simple git_vsts_close which requires two parameters the name of the project and the name of the repository. (in gitweb we would reference the namespace as vsts's project and the project would be vsts's repository). For the time being I'd like to take either the ssh url or the two parameters and do all the git things. in brief this is what i have so far
function git_vsts_clone {
local projectName=$1
local repositoryName=$2
if MISSING_ARG "usage: git_vsts_clone <project name> <repositoryName>\n projectName must be provided\n repositoryName must be provided" $projectName; then return 1; fi;
if MISSING_ARG "usage: git_vsts_clone <project name> <repositoryName>\n projectName must be provided\n repositoryName must be provided" $repositoryName; then return 1; fi;
local gitServer="ssh://mycompany#vs-ssh.visualstudio.com:22/${projectName}/_ssh/${repositoryName}"
local clonePath="/c/git/${projectName}/${repositoryName}"
local user_name=${USER:-${USERNAME}}
if [ ! -d $clonePath ]; then
INFO "Cloning $gitServer"
git clone $gitServer $clonePath || { ERROR "ERROR cloning $gitServer"; return 1;}
pushd $clonePath
INFO "Updating Submodules (gsui)"
git submodule update --init
INFO "adding user fork ${user_name}"
git remote add $user_name $gitServer.$user_name
git fetch $user_name
popd
INFO "Opening $clonePath in vscode"
fi
code $clonePath
}
last time when I tried to parse a url in bash I struggled with the whole split an item into an array. so I decided i'd try to use ruby (since it has a easy split method) so i've tried things like
$ gitServer='ssh://mycompany#vs-ssh.visualstudio.com:22/someProject/_ssh/myRepo'
$ ruby -e "a = '$gitServer'; b=a.split('/'); p b"
["ssh:", "", "mycompany#vs-ssh.visualstudio.com:22", "someProject", "_ssh", "myRepo"]
$ foo=`ruby -e "a = '$gitServer'; b=a.split('/'); p b"`
$ echo "${c[3]}"
$ echo "${c[0]}"
["ssh:", "", "mycompany#vs-ssh.visualstudio.com:22", "someProject", "_ssh", "myRepo"]
so I dunno. I don't have to use ruby it just seemed like a easy solution... now not so much. So how can I get the project and repository name out of the url in either bash or bash using ruby?
Here is a way you can get the values into environment variables using Ruby:
Assuming you have a URL environment variable containing the git repo url, such as created by the line below:
export URL='ssh://mycompany#vs-ssh.visualstudio.com:22/someProject/_ssh/myRepo'
You can do the following to put your desired values into other environment variables:
export PROJECT=`ruby -e "puts ENV['URL'].split('/')[3]"`
export REPO_NAME=`ruby -e "puts ENV['URL'].split('/')[5]"`

Resources