i'm working on my cicd in gitlab.
I have set up many environment variables, most of which are standard string/numbers. What I've done is to prefix them with "APP_" so that I can properly export to my project during cicd only the required variables. I do it this way:
export | grep APP_ | sed -e 's/APP_//g' | sed -e 's/declare -x //g' > ./app/settings/.env
This will basically take all the environement variables with APP_, remove the APP_, and store all of them in a file in ./app/settings/.env
This works like a charm
Now I'd like to do something similar for my file environment variables. What I've done is creating with a "FILE_" prefix, so I'd like to:
create one file per environement variable starting with "FILE_", naming the file as the environment variable name (but without the prefix FILE_)
store the files in .app/settings/files
How should I do so ?
At the moment i'm doing one by one but this isn't what I'd like:
echo "$FILE_MY_CERTIFICATE" > "./app/settings/files/my_certificate"
P.S. For those experts in gitlab env variables, I'm doing like this because I'm unable to use the standard "file" environement variable feature integrated in gitlab. The variables aren't in my build project so I'd like to find a workaround to suit my needs.
I do it this way:
Forget your code. It's just:
declare -p "${!APP_#}" > ./app/settings/.env
If you really specifically want to really parse something like export, use env -0.
How should I do so ?
declare -p "${!FILE_#}" > ./app/settings/files
I believe this may be a XY question and you should instead rethink your algorithm to use arrays or associative arrays instead.
Related
The real case:
Every time I set up my environment, I'd like to check and – create if are not existing – certain environment variables. So, instead of doing it manually all the time, I thought it would be great if I can have a file which stores the environment name env_N and environment value env_V pairs. Amongst all text file formats, the yaml looks the simplest and the more natural to store that info.
So what I thought it would be great if I suck in the yaml file with my environmental variables using yq and create associative array ready to be iterated over by zsh foreach loop:
foreach entry in my_assoc_arr
do
check_and_create(entry.env_N, entry.env_V)
done
with the final result of:
$ echo $env_N1
env_V1
$ echo $env_N2
env_V2
$ echo $env_N3
env_V3
...
The problem I'm having is to get my yaml to associative array using yq in shell zsh script.
After applying each suggestion from the comments, I was unable to create my associative array from yaml with yq. I had errors from yq like bad syntax or script worked or not depends on whether I have #!/bin/zsh switched on or commented out.
I got impression that my task is simple, but somehow I cant achieve this.
What I'm doing wrong here?
PS: I'm using zsh on macOS
Why would you want to use Yaml for this, when it’s much easier to write a simple shell script for it?
Simply create a text file with the following line for each env var:
# Set $FOO to 'bar' if $FOO does not yet exist.
export ${FOO=bar}
Then source this file from your shell (or another script).
See https://zsh.sourceforge.io/Doc/Release/Expansion.html#Parameter-Expansion
I want to print out all the variables defined in the file (not environment variables), so that I can quickly locate the error. I thought of printing through echo, but this is not friendly, is there any easy way to achieve this?
For example is as follow:
var1=${VAR1:-"test1"}
var2=${VAR2:-"test2"}
var3=${VAR1:-"test3"}
var4=${VAR1:-"test4"}
print like below:
var1=test1
var2=modify // modified by environment var
var3=test3
var4=test4
I really appreciate any help with this.
In Bash you can:
# List all variables to a file named before
declare -p > before
# source the file
. the_file
# list all variables to a file named after
declare -p > after
# difference between variables before and after sourcing the file
diff before after
You can manipulate with env -i bash -c to get a clean environment.
The other way is just to write a parser for your file. Simple sed 's/=.*//' the_file will give you a list of all variable definitions.
Context:
I started writing this script for easily changing connections for my raspberry pi zero (Raspibian Lite as OS), this is because I always needed to edit the wpa-supplicant config file and decided to do something about it as it is a really portable pc.
How it works:
The core of the program is to create profiles in format of dictionaries to store the name and passwd and apply that profile when needed. The profiles are added to the script code itself. I made it like this, every time a new profile is created this 2 lines are generated with the profile name corresponded. For example:
declare -A profile1
profile1=( ["name"]="name" ["pass"]="pass")
Problem:
To apply that profile I put at terminal prompt "./script --use profile1" so my goal is that it gets the details of the profile desired.
When I write that by :
echo "${$2[name]}" it outputs me a " bad substitution" error.
Things I tried and checked:
Shebang is #!/bin/bash
I tried substituting the $2 in a string and trying to execute it but I dont get anything good.
Things to consider:
Here is the link to the script so you can test it yourself, there are some things are a bit more complex than the thing indicated in the post but I just simplified it.
https://github.com/gugeldot23/wpa_scrip
You need nameref variables if you want to address the profile array name by reference:
declare -n profile # nameref variable profile
profile="$2" # Fills-in the nameref from argument 2
# Address the nameref instead of echo "${$2[name]}"
echo "${profile[name]}"
See: gnu.org Bash Manual / Bash Builtins / declare:
-n
Give each name the nameref attribute, making it a name reference to another variable. That other variable is defined by the value of name. All references, assignments, and attribute modifications to name, except for those using or changing the -n attribute itself, are performed on the variable referenced by name’s value. The nameref attribute cannot be applied to array variables.
I have to introduce some templating over text configuration files (yaml, xml, json) that already contain bash-like syntax variables. I need to preserve existing bash-like variables untouched but substitute my ones. List is dynamic, variables should come from environment. Something like simple processor taking "$${MY_VAR}}" pattern but ignoring $MY_VAR. Preferably pure Bash or as small number of tooling required as possible.
Pattern could be $$(VAR) or anything that can be easily separated from ${VAR} and $VAR. The key limitation - it is intended for a docker container startup procedure injecting environment variables into provided service configuration templates and this way building this configuration. So something like Java or even Perl processing is not an option.
Does anybody have a simple approach?
I was using the following bash processing for such variable substitution where original files had no variables. But now I need something one step smarter.
# process input file ($1) placing output into ($2) with shell variables substitution.
process_file() {
set -e
eval "cat <<EOF
$(<$1)
EOF
" | cat > $2
}
Obvious clean solution that is too complex for Docker file because of number of packages needed:
perl -p -i -e 's/\$\{\{([^}]+)\}\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < test.json
This filters out ${{VAR}}, even better - only set ones.
I'm trying to come up with a way of parameterizing some config files for a docker image using env vars. A sample input file could look something like:
<database>
<host>${DB_HOST}</host>
<name>${DB_NAME}</name>
<user>${DB_USER}</user>
<pass>${DB_PASSWORD}</pass>
</database>
and I'd want the output to basically be exactly the same, but with the fields filled in with the corresponding environment variable value.
Is there a simple way to do this directly in bash? The system I'm using doesn't natively support env vars, so I need to write these config files by hand, but I need to inject different vars for each environment I'm in so I don't particularly want to actually maintain real files for each env.
This is that simple as :
cat<<EOF > new_config_file
<database>
<host>${DB_HOST}</host>
<name>${DB_NAME}</name>
<user>${DB_USER}</user>
<pass>${DB_PASSWORD}</pass>
</database>
EOF
ls -l new_config_file
you're looking for envsubst
$ envsubst < config_template > config_instance
Just another option.
My generated config files often require just a bit of logic + embedding environment variable contents. At some point I got tired of reinventing the wheel every time and stitched together a simple tool in Go called Templater (https://github.com/reertech/templater).
It's just a standalone zero dependency binary for most of the systems, so you can just download it and use Go templating language (https://golang.org/pkg/text/template/) like that:
#!/bin/bash
YEAR=1997 \
NAME=John \
SURNAME=Connor \
./templater -t - -o /tmp/hello.txt <<EOF
Hello, {{env "NAME"}} {{env "SURNAME"}}!
{{ if env "YEAR" | parseInt | eq 1997 }}
Have a nice Judgement Day!
{{ end }}
EOF
In case you're not comfortable with prebuilt binaries you can use Go and build one yourself.
Hope this helps you or someone else who cannot cut it with just environment substitution and doesn't feel like Bashing.