Duplicate of https://unix.stackexchange.com/questions/159253/decoding-url-encoding-percent-encoding, but SO does not understand url from different stack.
Inside a docker container, I use environment variables (that I do not manage initialy, so I cannot act on them) inside a bash script.
One of the environement variable has one hexa character, but the rest of the string is OK.
Original string is: toto#t.
It is recuperated as toto%40t in the script with the following command:
echo "$VAR"
Note the # that is transformed into %40
With "minimal Ubuntu 20" (Installation of additionnal tools is possible, but I want to avoid it), pure bash if possible, how can I format the variable to get the original format ?
Related
I have a docker-compose file with environment variables set like:
identity-api:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://0.0.0.0:80
- SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104
It's just an example from https://github.com/dotnet-architecture/eShopOnContainers/blob/dev/src/docker-compose.override.yml.
The issue: I replaced underscore symbol with dot symbol here: ESHOP_EXTERNAL_DNS_NAME_OR_IP -> ESHOP_EXTERNAL_DNS.NAME_OR_IP.
After that, If I try to build docker-compose project in VS, it will not work. The error looks like:
Error DT1001 Invalid interpolation format for "environment" option in service "identity-api":"SpaClient=http://${ESHOP_EXTERNAL_DNS.NAME_OR_IP}:5104"
docker-compose C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets 202
So the question: what is wrong with the variable name here? Is dot symbol supported or not? Or maybe it can be escaped somehow?
In general, environment variable names with dots in them are allowed at a technical level but aren't well supported in any context, and I'd choose a different name.
In Docker Compose, it looks like the regexps used to identify variable expansions only support ASCII letters, digits, and underscores. This suggests dots in environment variable names aren't supported, and there's no way to escape them.
I think variables with dots in them also aren't supported in shell scripts, but I'm having trouble proving that to myself reading the POSIX spec. A Name also only consists of ASCII letters, digits, and underscores; parameters either have a name, a number, or a single symbol; and variables are parameters with names; but none of the formal description of this says a "name" is a Name.
One easy thing to demonstrate is running the bash shell in the ubuntu Docker image. We can use the docker run -e option to set arbitrary environment variables (that don't contain =) but we can't really use any shell features to expand variables with dots:
host$ docker run --rm -it -e foo=A -e foo.bar=B ubuntu
# Demonstrate the environment variable is set:
root#227bec28c674:/# env | grep foo
foo.bar=B
foo=A
# It won't get expanded without braces:
root#227bec28c674:/# echo $foo.bar
A.bar
# With braces, it's considered illegal syntax:
root#227bec28c674:/# echo ${foo.bar}
bash: ${foo.bar}: bad substitution
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
I use bash to configure many of my build tools.
I want to use variables to set certain things so that the build tools can be used in many environments.
Currently I do
exports var=val
and then I do
$var
when I need to use it.
Is this the best way to go about it, as I know there are many ways to do things in bash.
**Example**
#!/bin/bash
path_bash="$HOME/root/config/bash/"
source "${path_bash}_private.sh"
source "${path_bash}config.sh"
source "${path_bash}utility.sh"
source "${path_bash}workflow.sh"
source "${path_bash}net.sh"
source "${path_bash}makeHTM.sh"
#
#
#
# Divider - Commands
#
#
#
cd ~/root
Skip the export unless you really need it (that is, unless you need that variable to propagate to unrelated (=execed) processes that you execute).
If you do export, it's usually a good idea to capitalize the variable (export VAR=val) to make it apparent that it will spread to executed binaries.
When you refer to a shell variable, you usually want to double quote it ("$var") unless you need glob expansion and whitespace-splitting (splitting on $IFS characters, to be exact) done on that variable expansion.
sparrow automation framework gives you an opportunity to configure bash scripts in many ways :
passing command line named parameters
configuration files in yaml format
configuration files in json format
configuration files in config::general format
Of course you need to pack your bash scripts into sparrow plugins first to get this behavior , but this is not a big deal . Follow https://github.com/melezhik/sparrow#configuring-tasks for details .
PS. disclosure - I am the sparrow tool author .
I am working on a shell script to retrieve variable content from a JSON file via JQ. The JSON file is in string format (no matter whether this is a real string or a number) and to retrieve the variable in my bash script I did something like this
my_domain=$(cat /vagrant/data_bags/config.json | jq ."app"[0]."domain")
The above code once echoed results in "mydomain" with a beginning and a trailing quote sign. I though this was a normal behaviour of the echo command. However, while concatenating my variable with another shell command the system raise an error. For instance, the following command
cp /vagrant/public_html/index.php "/var/www/"+$my_domain+"/index.php"
fails with the following error
cp: cannot create regular file `/var/www/+"mydomain"+/index.php': No such file or directory
At this stage, I wasn't able to identify whether it's me doing the wrong concatenation with the plus sign or the variable is effectively including the quotes that in any case will end up generating an error.
I have tried to replace the quotes in my variable, but I ended up getting the system raising a "Command not found" error.
Can somebody suggest what am I doing wrong?
+ is not used for string concatenation in bash (or perl, or php). Just:
cp /vagrant/public_html/index.php "/var/www/$my_domain/index.php"
Embedding a variable inside a double-quoted text string is known as interpolation, and is one of the reasons why we need the $ prefix, to indicate that this is a variable. Interpolation is specifically not done inside single quoted strings.
Braces ${my_domain} are not required because the / directory separators are not valid characters in a variable name, so there is no ambiguity.
For example:
var='thing'
echo "Give me your ${var}s" # Correct, appends an 's' after 'thing'
echo "Give me your $vars" # incorrect, looks for a variable called vars.
If a variable (like 'vars') does not exist then (by default) it will not complain, it will just give an empty string. Braces (graph brackets) are required more in c-shell (csh or tcsh) because of additional syntax for modifying variables, which involves special trailing characters.
You don't need to use + to concatenate string in bash, change your command to
cp /vagrant/public_html/index.php "/var/www/"${my_domain}"/index.php"
My problem was not related only to the wrong concatenation, but also to the JQ library that after parsing the value from the JSon file was returning text between quotes.
In order to avoid JQ doing this, just add the -rawoutput parameter when calling JQ.
The other day I stumbled upon a question on SO. If I wanted to extract the value of HOSTNAME in /etc/sysconfig/network which contains
NETWORKING=yes
HOSTNAME=foo
now I can do grep and cut to get the foo but there was some bash magic involved for a similar issue. I don't know what to search for that and I can't seem to find the question now. it involved something like #{HOSTNAME} . As if it was treating HOSTNAME as a key and foo as a value.
If that configuration file is compatible with shell syntax, simply include it as a shell script. IIRC the files in /etc/sysconfig on Red Hat-like distributions are indeed designed to be parsable by a shell. Note that this means that
If shell special characters may end up in a variable's value, they must be properly quoted. For example, var="value with spaces" requires the quotes. var="with\$dollar" requires the backslash.
The script may run arbitrary code that will be executed, so this is only ok if you trust its content.
If these assumptions are valid, then you can go the simple route:
. /etc/sysconfig/network
echo "$HOSTNAME"
Regarding the quoting and braces, see $VAR vs ${VAR} and to quote or not to quote.