I have a docker-compose file with environment variables set like:
identity-api:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://0.0.0.0:80
- SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104
It's just an example from https://github.com/dotnet-architecture/eShopOnContainers/blob/dev/src/docker-compose.override.yml.
The issue: I replaced underscore symbol with dot symbol here: ESHOP_EXTERNAL_DNS_NAME_OR_IP -> ESHOP_EXTERNAL_DNS.NAME_OR_IP.
After that, If I try to build docker-compose project in VS, it will not work. The error looks like:
Error DT1001 Invalid interpolation format for "environment" option in service "identity-api":"SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104"
docker-compose C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets 202
So the question: what is wrong with the variable name here? Is dot symbol supported or not? Or maybe it can be escaped somehow?
In general, environment variable names with dots in them are allowed at a technical level but aren't well supported in any context, and I'd choose a different name.
In Docker Compose, it looks like the regexps used to identify variable expansions only support ASCII letters, digits, and underscores. This suggests dots in environment variable names aren't supported, and there's no way to escape them.
I think variables with dots in them also aren't supported in shell scripts, but I'm having trouble proving that to myself reading the POSIX spec. A Name also only consists of ASCII letters, digits, and underscores; parameters either have a name, a number, or a single symbol; and variables are parameters with names; but none of the formal description of this says a "name" is a Name.
One easy thing to demonstrate is running the bash shell in the ubuntu Docker image. We can use the docker run -e option to set arbitrary environment variables (that don't contain =) but we can't really use any shell features to expand variables with dots:
host$ docker run --rm -it -e foo=A -e foo.bar=B ubuntu
# Demonstrate the environment variable is set:
root#227bec28c674:/# env | grep foo
foo.bar=B
foo=A
# It won't get expanded without braces:
root#227bec28c674:/# echo $foo.bar
A.bar
# With braces, it's considered illegal syntax:
root#227bec28c674:/# echo ${foo.bar}
bash: ${foo.bar}: bad substitution
Related
I have a situation where I have an environment variable with space character inside. Some tools do not like quoting the value of the variable, as they will treat the quote as part of the variable.
This is set in a .env file.
PIP_EXTRA_INDEX_URL="https://token#repo https://token#repo"
When I include and export this .env file in a Makefile, I get this warning.
WARNING: Location '"https://token#repo' is ignored:
it is either a non-existing path or lacks a specific scheme.
But I have seen this behavior as initially mentioned also with other tools. Is there a way to handle this?
In the Makefile, I include it like below.
include .env
export
build:
docker build --build-arg PIP_EXTRA_INDEX_URL -t myimage .
Makefiles are not shell scripts and it is not possible to use the same syntax to define variables in both the shell and in make, except in very limited situations.
In the shell, you can have multiple assignments on the same line or even run programs on the same line. So, if your assignment has whitespace in it you have to quote it as you've done here.
In make, the syntax of an assignment is that all text after the assignment (and leading whitespace) becomes the value of the variable and there is no quoting needed; any quotes that are seen are kept as part of the variable value.
So, in the shell this assignment:
PIP_EXTRA_INDEX_URL='https://token#repo https://token#repo'
sets the shell variable PIP_EXTRA_INDEX_URL to the value https://token#repo https://token#repo ... note the quotes are stripped from the value by the shell.
In make this assignment:
PIP_EXTRA_INDEX_URL='https://token#repo https://token#repo'
sets the shell variable PIP_EXTRA_INDEX_URL to the value 'https://token#repo https://token#repo' ... note the quotes are not stripped from the value by make.
So if you use this value in a recipe like this:
do something "$(PIP_EXTRA_INDEX_URL)"
then make will expand that variable and you'll get:
do something "'https://token#repo https://token#repo'"
(including quotes) and that's your problem.
It works like this.
build:
docker build --build-arg PIP_EXTRA_INDEX_URL=$(PIP_EXTRA_INDEX_URL) -t myimage .
Duplicate of https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding, but SO does not understand url from different stack.
Inside a docker container, I use environment variables (that I do not manage initialy, so I cannot act on them) inside a bash script.
One of the environement variable has one hexa character, but the rest of the string is OK.
Original string is: toto#t.
It is recuperated as toto%40t in the script with the following command:
echo "$VAR"
Note the # that is transformed into %40
With "minimal Ubuntu 20" (Installation of additionnal tools is possible, but I want to avoid it), pure bash if possible, how can I format the variable to get the original format ?
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
On a CentOS 5 host (with bash 3.2.32), we use Ruby (1.8.7) to
ENV['AWS_foo-bar_ACCESS_KEY'] = xxxxx
Then, using bash, we run a shell script that does:
BUCKET_NAME=$1
AWS_ACCESS_KEY_ID_VAR="AWS_${BUCKET_NAME}_ACCESS_KEY_ID"
AWS_ACCESS_KEY_ID="${!AWS_ACCESS_KEY_ID_VAR}"
export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
This works fine on CentOS 5.
However, on CentOS 6 (with bash 4.1.2), we get the error
-bash: export: `AWS_foo-bar_ACCESS_KEY_ID=xxxxx': not a valid identifier
It is our understanding that this fails because - is not allowed in the variable name. But why does this work on bash 3.2 and not bash 4.1?
The "why" is almost irrelevant: The POSIX standard makes it very clear that export is only required to support arguments which are valid names, and anything with a dash is not a valid name. Thus, no POSIX shell is required to support exporting or expanding variable names with dashes, via indirect expansion or otherwise.
It's worth noting that ShellShock -- a major security bug caused by sloppy handling of environment contents -- is fixed in the bash 4.1 present in the current CentOS 6 updates repo; increased rigor in an area which spawned security bugs should be no surprise.
The remainder of this answer will focus on demonstrating that the new behavior of bash 4.1 is explicitly allowed, or even required, by POSIX -- and thus that the prior behavior was an undefined implementation artifact.
To quote POSIX on environment variables:
These strings have the form name=value; names shall not contain the character '='. For values to be portable across systems conforming to IEEE Std 1003.1-2001, the value shall be composed of characters from the portable character set (except NUL and as indicated below). There is no meaning associated with the order of strings in the environment. If more than one string in a process' environment has the same name, the consequences are undefined.
Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore) from the characters defined in Portable Character Set and do not begin with a digit. Other characters may be permitted by an implementation; applications shall tolerate the presence of such names. Uppercase and lowercase letters shall retain their unique identities and shall not be folded together. The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities.
Note: Other applications may have difficulty dealing with environment variable names that start with a digit. For this reason, use of such names is not recommended anywhere.
Thus:
Tools (including the shell) are required to fully support environment variable names with uppercase and lowercase letters, digits (except in the first position), and the underscore.
Tools (including the shell) may modify their behavior based on environment variables with names that comply with the above and additionally do not contain lowercase letters.
Tools (including the shell) should tolerate other names -- meaning they shouldn't crash or misbehave in their presence -- but are not required to support them.
Finally, shells are explicitly allowed to discard environment variable names which are not also shell variable names. From the relevant standard:
It is unspecified whether environment variables that were passed to the shell when it was invoked, but were not used to initialize shell variables (see Shell Variables) because they had invalid names, are included in the environment passed to execl() and (if execl() fails as described above) to the new shell.
Moreover, what defines a valid shell name is well-defined:
Name - In the shell command language, a word consisting solely of underscores, digits, and alphabetics from the portable character set. The first character of a name is not a digit.
Notably, only underscores (not dashes) are considered part of a valid name in a POSIX-compliant shell.
...and the POSIX specification for export explicitly uses the word "name" (which it defined in the text quoted above), and describes it as applying to "variables" (shell variables, the restrictions on names for which are also subject to restrictions quoted elsewhere in this document):
The shell shall give the export attribute to the variables corresponding to the specified names, which shall cause them to be in the environment of subsequently executed commands. If the name of a variable is followed by = word, then the value of that variable shall be set to word.
All the above being said -- if your operating system provides a /proc/self/environ which represents the state of your enviroment variables at process startup (before a shell has, as it's allowed to do, potentially discarded any variables which don't have valid names in shell), you can extract content with invalid names like so:
# using a lower-case name where possible is in line with POSIX guidelines, see above
aws_access_key_id_var="AWS_${BUCKET_NAME}_ACCESS_KEY_ID"
while IFS= read -r -d '' var; do
[[ $var = "$aws_access_key_id_var"=* ]] || continue
val=${var#"${aws_access_key_id_var}="}
break
done </proc/self/environ
echo "Extracted value: $val"
The other day I stumbled upon a question on SO. If I wanted to extract the value of HOSTNAME in /etc/sysconfig/network which contains
NETWORKING=yes
HOSTNAME=foo
now I can do grep and cut to get the foo but there was some bash magic involved for a similar issue. I don't know what to search for that and I can't seem to find the question now. it involved something like #{HOSTNAME} . As if it was treating HOSTNAME as a key and foo as a value.
If that configuration file is compatible with shell syntax, simply include it as a shell script. IIRC the files in /etc/sysconfig on Red Hat-like distributions are indeed designed to be parsable by a shell. Note that this means that
If shell special characters may end up in a variable's value, they must be properly quoted. For example, var="value with spaces" requires the quotes. var="with\$dollar" requires the backslash.
The script may run arbitrary code that will be executed, so this is only ok if you trust its content.
If these assumptions are valid, then you can go the simple route:
. /etc/sysconfig/network
echo "$HOSTNAME"
Regarding the quoting and braces, see $VAR vs ${VAR} and to quote or not to quote.