I have to introduce some templating over text configuration files (yaml, xml, json) that already contain bash-like syntax variables. I need to preserve existing bash-like variables untouched but substitute my ones. List is dynamic, variables should come from environment. Something like simple processor taking "$${MY_VAR}}" pattern but ignoring $MY_VAR. Preferably pure Bash or as small number of tooling required as possible.
Pattern could be $$(VAR) or anything that can be easily separated from ${VAR} and $VAR. The key limitation - it is intended for a docker container startup procedure injecting environment variables into provided service configuration templates and this way building this configuration. So something like Java or even Perl processing is not an option.
Does anybody have a simple approach?
I was using the following bash processing for such variable substitution where original files had no variables. But now I need something one step smarter.
# process input file ($1) placing output into ($2) with shell variables substitution.
process_file() {
set -e
eval "cat <<EOF
$(<$1)
EOF
" | cat > $2
}
Obvious clean solution that is too complex for Docker file because of number of packages needed:
perl -p -i -e 's/\$\{\{([^}]+)\}\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < test.json
This filters out ${{VAR}}, even better - only set ones.
Related
For example,
# Execute the pre-hook.
export SHELL=#shell#
param1=#param1#
param2=#param2#
param3=#param3#
param4=#param4#
param5=#param5#
if test -n "#preHook#"; then
. #preHook#
fi
For context, this is from a shell script in a commit from 2004 in the Nixpkgs repo; tried to see if this maybe a reference feature but string "shell" only occurs once (in a case-sensitive search) in the entire file.
The answer by Chris Dodd is correct, insofar as there's no intrinsic meaning to the shell -- and #foo# is thus commonly used as a sigil. Insofar as you encountered this in nixpkgs, it provides some stdenv tools specifically for implementing this pattern.
As documented at https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-functions, nixpkgs stdenv provides shell functions including substitute, substituteAll, substituteInPlace &c. which will replace #foo# values with the content of corresponding variables.
In the context of the linked commit, subsitutions of that form can be seen being performed in pkgs/build-wrapper/gcc-wrapper/builder.sh:
sed \
-e "s^#gcc#^$src^g" \
-e "s^#out#^$out^g" \
-e "s^#bash#^$SHELL^g" \
-e "s^#shell#^$shell^g" \
< $gccWrapper > $dst
...is replacing #out# with the value of $out, #bash# with the value of $SHELL, etc.
The # symbol has no meaning to the shell -- it is a punctuation character that will pretty much never occur in any actual shell script.
This makes it a good choice to use for patterns in script templates -- the basic idea being that a simple search-and-replace process will be used (perhaps with a sed script as in the link you show) to rewrite the template into an actual shell script. Every string of the form #name# in the template will be replaced by some other string related to the environment in which the script is being installed.
i'm working on my cicd in gitlab.
I have set up many environment variables, most of which are standard string/numbers. What I've done is to prefix them with "APP_" so that I can properly export to my project during cicd only the required variables. I do it this way:
export | grep APP_ | sed -e 's/APP_//g' | sed -e 's/declare -x //g' > ./app/settings/.env
This will basically take all the environement variables with APP_, remove the APP_, and store all of them in a file in ./app/settings/.env
This works like a charm
Now I'd like to do something similar for my file environment variables. What I've done is creating with a "FILE_" prefix, so I'd like to:
create one file per environement variable starting with "FILE_", naming the file as the environment variable name (but without the prefix FILE_)
store the files in .app/settings/files
How should I do so ?
At the moment i'm doing one by one but this isn't what I'd like:
echo "$FILE_MY_CERTIFICATE" > "./app/settings/files/my_certificate"
P.S. For those experts in gitlab env variables, I'm doing like this because I'm unable to use the standard "file" environement variable feature integrated in gitlab. The variables aren't in my build project so I'd like to find a workaround to suit my needs.
I do it this way:
Forget your code. It's just:
declare -p "${!APP_#}" > ./app/settings/.env
If you really specifically want to really parse something like export, use env -0.
How should I do so ?
declare -p "${!FILE_#}" > ./app/settings/files
I believe this may be a XY question and you should instead rethink your algorithm to use arrays or associative arrays instead.
I am attempting to add a parameter based on an additional list 'list2.txt' that I have created and I am not quite sure how to implement it.
My running code
while read i
do
sed "s/Pie/$i/g" old_script.sh > new_script.$i.sh
sbatch new_script.$i.sh
done<list.txt
But I want to add the following condition with based on a new list... and I am not quite sure how to implement it into my working script
sed "s/Apple/__/g"
sed allows several ways to supply multiple commands. You can give them individually with -e or just write them into a single script string.
GNU sed allows commands on the same line to be separated with semicolons, and is genrally what you will find, but if you don't have that version you can use embedded newlines. As long as it's quoted it will work fine.
sed "s/Pie/$i/g; s/Apple/__/g;" old_script.sh # GNU specific but common
or
sed "
s/Pie/$i/g
s/Apple/__/g
" old_script.sh # general, should always work.
These are both valid.
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
I'm dealing with a pipeline of predominantly shell and Perl files, all of which pass parameters (paths) to the next. I decided it would be better to use a single file to store all the paths and just call that for every file. The issue is I am using awk to grab the files at the beginning of each file, and it's turning out to be a lot of repetition.
My question is: I do not know if there is a way to store key-value pairs in a file so shell can natively do something with the key and return the value? It needs to access an external file, because the pipeline uses many scripts and a map in a specific file would result in parameters being passed everywhere. Is there some little quirk I do not know of that performs a map function on an external file?
You can make a file of env var assignments and source that file as need, ie.
$ cat myEnvFile
path1=/x/y/z
path2=/w/xy
path3=/r/s/t
otherOpt1="-x"
Inside your script you can source with either . myEnvFile or the more versbose version of the same feature sourc myEnvFile (assuming bash shell) , i.e.
$cat myScript
#!/bin/bash
. /path/to/myEnvFile
# main logic below
....
# references to defined var
if [[ -d $path2 ]] ; then
cd $path2
else
echo "no pa4h2=$path2 found, can't continue" 1>&1
exit 1
fi
Based on how you've described your problem this should work well, and provide a-one-stop-shop for all of your variable settings.
IHTH
In bash, there's mapfile, but that reads the lines of a file into a numerically-indexed array. To read a whitespace-separated file into an associative array, I would
declare -A map
while read key value; do
map[$key]=$value
done < filename
However this sounds like an XY problem. Can you give us an example (in code) of what you're actually doing? When I see long piplines of grep|awk|sed, there's usually a way to simplify. For example, is passing data by parameters better than passing via stdout|stdin?
In other words, I'm questioning your statement "I decided it would be better..."