I have a situation where I have an environment variable with space character inside. Some tools do not like quoting the value of the variable, as they will treat the quote as part of the variable.
This is set in a .env file.
PIP_EXTRA_INDEX_URL="https://token#repo https://token#repo"
When I include and export this .env file in a Makefile, I get this warning.
WARNING: Location '"https://token#repo' is ignored:
it is either a non-existing path or lacks a specific scheme.
But I have seen this behavior as initially mentioned also with other tools. Is there a way to handle this?
In the Makefile, I include it like below.
include .env
export
build:
docker build --build-arg PIP_EXTRA_INDEX_URL -t myimage .
Makefiles are not shell scripts and it is not possible to use the same syntax to define variables in both the shell and in make, except in very limited situations.
In the shell, you can have multiple assignments on the same line or even run programs on the same line. So, if your assignment has whitespace in it you have to quote it as you've done here.
In make, the syntax of an assignment is that all text after the assignment (and leading whitespace) becomes the value of the variable and there is no quoting needed; any quotes that are seen are kept as part of the variable value.
So, in the shell this assignment:
PIP_EXTRA_INDEX_URL='https://token#repo https://token#repo'
sets the shell variable PIP_EXTRA_INDEX_URL to the value https://token#repo https://token#repo ... note the quotes are stripped from the value by the shell.
In make this assignment:
PIP_EXTRA_INDEX_URL='https://token#repo https://token#repo'
sets the shell variable PIP_EXTRA_INDEX_URL to the value 'https://token#repo https://token#repo' ... note the quotes are not stripped from the value by make.
So if you use this value in a recipe like this:
do something "$(PIP_EXTRA_INDEX_URL)"
then make will expand that variable and you'll get:
do something "'https://token#repo https://token#repo'"
(including quotes) and that's your problem.
It works like this.
build:
docker build --build-arg PIP_EXTRA_INDEX_URL=$(PIP_EXTRA_INDEX_URL) -t myimage .
Related
I have a docker-compose file with environment variables set like:
identity-api:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://0.0.0.0:80
- SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104
It's just an example from https://github.com/dotnet-architecture/eShopOnContainers/blob/dev/src/docker-compose.override.yml.
The issue: I replaced underscore symbol with dot symbol here: ESHOP_EXTERNAL_DNS_NAME_OR_IP -> ESHOP_EXTERNAL_DNS.NAME_OR_IP.
After that, If I try to build docker-compose project in VS, it will not work. The error looks like:
Error DT1001 Invalid interpolation format for "environment" option in service "identity-api":"SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104"
docker-compose C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets 202
So the question: what is wrong with the variable name here? Is dot symbol supported or not? Or maybe it can be escaped somehow?
In general, environment variable names with dots in them are allowed at a technical level but aren't well supported in any context, and I'd choose a different name.
In Docker Compose, it looks like the regexps used to identify variable expansions only support ASCII letters, digits, and underscores. This suggests dots in environment variable names aren't supported, and there's no way to escape them.
I think variables with dots in them also aren't supported in shell scripts, but I'm having trouble proving that to myself reading the POSIX spec. A Name also only consists of ASCII letters, digits, and underscores; parameters either have a name, a number, or a single symbol; and variables are parameters with names; but none of the formal description of this says a "name" is a Name.
One easy thing to demonstrate is running the bash shell in the ubuntu Docker image. We can use the docker run -e option to set arbitrary environment variables (that don't contain =) but we can't really use any shell features to expand variables with dots:
host$ docker run --rm -it -e foo=A -e foo.bar=B ubuntu
# Demonstrate the environment variable is set:
root#227bec28c674:/# env | grep foo
foo.bar=B
foo=A
# It won't get expanded without braces:
root#227bec28c674:/# echo $foo.bar
A.bar
# With braces, it's considered illegal syntax:
root#227bec28c674:/# echo ${foo.bar}
bash: ${foo.bar}: bad substitution
I have a text file called OPTIONS.txt storing all flags of Makefile:
arg1=foo arg2="-foo -bar"
I want to pass all flags in this file to make. However,
make `cat OPTIONS.txt`
fails with make: invalid option -- 'a'. It seems that shell interprets it as:
make arg1=foo arg2="-foo -bar"
^argv[1] ^argv[2] ^argv[3]
Is there any way to make it interpreted as:
make arg1=foo arg2="-foo -bar"
^argv[1] ^--------argv[2]
Since you control the options file, store the options one per line:
arg1=foo
arg2="-foo -bar"
Then in the shell, you'll read the file into an array, one element per line:
readarray -t opts < OPTIONS.txt
Now you can invoke make and keep the options whole:
make "${opts[#]}"
If you want the shell to interpret quotes after backtick expansion you need to use eval, like this:
eval make `cat OPTIONS.txt`
however just be aware that this evaluates everything, so if you have quoted content outside of the backticks you'll get the same issue:
eval make `cat OPTIONS.txt` arg4="one two"
will give an error. You'd have to double-quote the arg4, something like this:
eval make `cat OPTIONS.txt` arg4='"one two"'
In general it's tricky to do stuff like this from the command line, outside of scripts.
ETA
The real problem here is that we don't have a set of requirements. Why do you want to put these into a file, and what kind of things are you adding; are they only makefile variable assignments, or are there other make options here as well such as -k or similar?
IF the OP controls (can change) the format of the file AND the file contains content only used by make AND the OP doesn't care about the variables being command line assignments vs. regular assignments AND there are only variable assignments and not other options, then they can just (a) put each variable assignment on its own line, (b) remove all quotes, and (c) use include OPTIONS.txt from inside the makefile to "import" them.
I read in http://wiki.bash-hackers.org/howto/conffile that it is recommended to put double quotes around values in BASH configuration files which are to be sourced into a script.
The file to be sourced should be formated in key="value" format,
otherwise bash will try to interpret commands
However, I am not aware of any difference in BASH's behavior when sourcing a configuration file if the value has double quotes or not, assuming there is no whitespace in the value. I'm sure there are some more complex cases where the double quotes are vital (e.g. using other variables as the value), but for the simple cases below, would double quotes cause BASH to behave any differently, even if the difference is only behind-the-scenes? I'm wondering if the first configuration file below could cause BASH to search for a named foobar before assigning it as a string, but from my testing it doesn't appear to do so.
# Configuration file 1
myDir=/var/tmp/test/
myString=foobar
myInteger=20
# Configuration file 2
myDir="/var/tmp/test/"
myString="foobar"
myInteger="20"
source configurationFile1
echo "$myDir"
echo "$myString"
echo "$myInteger"
source configurationFile2
echo "$myDir"
echo "$myString"
echo "$myInteger"
It's a style issue. In the examples you show, the quotes aren't strictly necessary. myDir=/var/tmp/text and myDir="/var/tmp/text" do exactly the same thing. Other values may require quotes to make the assignment correct.
The allusion is to the fact that these aren't really configuration files; they're just bash scripts that are intended to contain only assignments. Something like
foo=bar baz
is not an assignment; it's a simple command that tries to run baz with a variable named foo in its environment. Here, quotes are required:
foo="bar baz"
to make a proper assignment, in contrast to other "actual" configuration file formats where everything following the = (and optionally some post-= whitespace`) is considered part of the value to be assigned.
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
I have a bash script which imports same variables from another static file, which itself uses environment variables set by another file when the script is invoked.
This is the file that gets imported and sets some variables.
# package.mk
PKG_NAME="binutils"
PKG_VERSION="2.24"
PKG_URL_TYPE="http"
PKG_URL="http://ftp.gnu.org/gnu/binutils/${PKG_NAME}-${PKG_VERSION}.tar.bz2"
PKG_DEPENDS=""
PKG_SECTION="devel"
PKG_CONFIGURE_OPTS="--prefix=${TOOLS} \
--target=${TARGET} \
--with-sysroot=${TOOLS}/${TARGET} \
--disable-nls \
--disable-multilib"
It is used by the builds script as so:
#!/bin/bash
# Binutils
. settings/config
pkg_dir="$(locate_package 'binutils')"
. "${pkg_dir}/package.mk"
# etc...
"${CLFS_SOURCES}/${PKG_NAME}-${PKG_VERSION}/configure" "${PKG_CONFIGURE_OPTS}"
# etc...
This script first imports the settings/config file which has a bunch of global variables used by this script and others, and exports them so they are available as environment variables. It then locates the correct package.mk file for the specific component we are building, and imports it as well. So far, so good.
However when I double-quote the options (PKG_CONFIGURE_OPTS) for the configure script:
"${CLFS_SOURCES}/${PKG_NAME}-${PKG_VERSION}/configure" "${PKG_CONFIGURE_OPTS}"`
I get the following error:
gcc: error: unrecognized command line option ‘--with-sysroot=/root/LiLi/target/cross-tools/arm-linux-musleabihf’
If I leave it not quoted like:
"${CLFS_SOURCES}/${PKG_NAME}-${PKG_VERSION}/configure" ${PKG_CONFIGURE_OPTS}`
it works fine (--with-sysroot= is indeed a valid configure flag for binutils).
Why is this? What can I change so that I can double-quote that portion (going by the bash wisdom that one should double-quote just about everything).
Quoting the variable means the entire thing is passed as a single argument, spaces and newlines included. You want word splitting to be performed so that the string is treated as multiple arguments. That's why leaving it unquoted works.
If you're looking for the "right" way to handle this, I recommend using an array. An array can hold multiple values while also preserving whitespace properly.
PKG_CONFIGURE_OPTS=(--prefix="$TOOLS"
--target="$TARGET"
--with-sysroot="$TOOLS/$TARGET"
--disable-nls
--disable-multilib)
...
"$CLFS_SOURCES/$PKG_NAME-$PKG_VERSION/configure" "${PKG_CONFIGURE_OPTS[#]}"