Environment variables with multiline values may confuse env's output:
# export A="B
> C=D
> E=F"
# env
A=B
C=D
E=F
TERM=xterm
SHELL=/bin/bash
USER=root
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/root
LANG=en_US.UTF-8
PS1=\h:\w\$
SHLVL=1
HOME=/root
LOGNAME=root
_=/usr/bin/env
In this case I can not just use awk -F= to extract all names because it shows wrong names C and E:
# env | awk -F= '{print $1}'
A
C
E
TERM
SHELL
USER
MAIL
PATH
PWD
LANG
PS1
SHLVL
HOME
LOGNAME
_
Then I figured out that env supports flag -0 to end each output line with 0 byte rather than newline, so using sed I could cut off the values in bash:
# env -0 | sed -e ':a;N;$!ba;s/\([^=]\+\)=\([^\x00]*\)\x00/\1\n/g'
A
TERM
SHELL
USER
MAIL
PATH
PWD
LANG
PS1
SHLVL
HOME
LOGNAME
_
But BusyBox's version of env does not support flag -0. Is there another way to do it?
If you are using linux (I thought busybox ran only on linux, but I may be wrong), /proc/self/environ contains a NUL separated environment in the same form as env -0 gives you. You can replace env -0 | with < /proc/self/environ.
sed -e ':a;N;$!ba;s/\([^=]\+\)=\([^\x00]*\)\x00/\1\n/g' < /proc/self/environ
This is maybe not an elegant but working solution. It first extracts all possible names from env's output, then verifies each of them using shell's expansion ${parameter:+word}. And finally it removes duplicates, since the same variable name could be printed on several lines in env's output (as a real variable name and as a part of some other variable's multiline value):
env | awk -F= '/[a-zA-Z_][a-zA-Z_0-9]*=/ {
if (!system("[ -n \"${" $1 "+y}\" ]")) print $1 }' | sort | uniq
PS: The | sort | uniq part can be also implemented in awk.
This will break if your environment variable values contain nulls. But that would also break from POSIX compatibility.
So it should work.
...unless you expect to encounter environment variable names which contain newlines. In that case the newlines will be truncated when they're displayed. However I can't seem to fathom how to create an environment variable with a newline in it in a busybox shell. My local shells balk at it at any rate. So I don't think that would be a big problem. As far as POSIX says, Other characters may be permitted by an implementation; applications shall tolerate the presence of such names. so I think stripping them and not erroring-out is tolerable.
# Read our environment; it's delimited by null bytes.
# Remove newlines
# Replace null bytes with newlines
# On each line, grab everything before the first '='
cat /proc/self/environ | tr -d '\n' | tr '\0' '\n' | cut -d '=' -f 1
Related
I have a .env file,
# Some variables
USERNAME=user1
PASSWORD=
# Other variables
URL=https://example.com
TOKEN=muDm8qJ6mqM__YX024dk1RXluPnwd-3mxyt1LwLoI4ISiVPA==
Purpose
My purpose is to create one line bash command to print each non-empty variables in <name> <value> format, which shall
Print one line for each variable with [space] between name and
value;
Ignore comments;
Ignore variables with no value;
Expected output
The expected output is,
USERNAME user1
URL https://example.com
TOKEN muDm8qJ6mqM__YX024dk1RXluPnwd-3mxyt1LwLoI4ISiVPA==
Current Solution
My current solution is as following,
grep -v '^#' .env | grep -v '^[[:space:]]*$' | grep -v '=$' | sed 's/=/ /' | while read -r line; do echo $line; done
Actual output
USERNAME user1
URL https://example.com
It only prints out the first two lines, the issue with last line is caused by the equal (=) signs in TOKEN value.
Help needed
Anyone can help me to rectify the command? also welcome if there is easier way to achieve the goal.
Using sed
$ sed '/=\</!d;s/=/ /' input_file
USERNAME user1
URL https://example.com
TOKEN muDm8qJ6mqM__YX024dk1RXluPnwd-3mxyt1LwLoI4ISiVPA==
Delete lines that do not match a valid variable i.e have no inpurt after =, this will also match blank lines and for your example data, lines with comments as there is no valid variable on the line. Finally, replace the first occurance of the = equal character for a space.
The only way to process any environment variable file is to actually interpret it. Various methods for printing out the variable lists are described below.
Various “environment files” are usually designed to be run by POSIX shell to evaluate all the variables. That makes it possible to call external commands, perform calculations etc. when assigning to the variables.
Do this only is you trust you input file. Any command can be executed by parsing the file using the script below.
You can source the file (builtin command .):
. .env
After you source it, you can evaluate the variables. If you know what all variables you want to print, use this (for vars USERNAME and PASSWORD)
. .env
for var_name in USERNAME PASSWORD
do
printf "%s %s\n" "$var_name" "${!var_name}"
done
If you want to print all the variables that are explicitly specified, use this approach:
. .env
grep '^\w+=' | sed 's/=//' | while read var_name
do
printf "%s %s\n" "$var_name" "${!var_name}"
done
But this solution is also not perfect, because the .env file can use constructions different from var_name=value to assign the vars. Use the set command to print all variables.
This prints out absolutely all variables. (In reality, the environment file should inherit its variables from the system environment vars, so there is a lot of them.)
. .env
printenv | sed 's/=/ /'
The sed is used to get rid of the equals signs.
Beware of some dangerous variable values. The variables can contain even newline character, backspaces and other control characters.
There is also a lot of various utilities for printing the variable in a way that allow loading them back to the shell. For example:
declare -p or typeset -p — prints out declare commands to declare all the currently set shell variables
export -p — the same, but for environment variables (that is probably that thing you want)
set — prints out all shell variables
If the file contains only comments and simple key=value lines without # chars in strings, you can use this pipeline:
sed 's/#.*//' | fgrep = | sed 's/=/ /'
Or if you do not want to output empty variables, use this:
sed 's/#.*//' | grep =. | sed 's/=/ /'
I couldn't find an existing answer to this specific case: I would like to simply display all exported environment variables sorted by their name. Normally I can do this like simply like:
$ env | sort
However, if some environment variables contain newlines in their values (as is the case on the CI system I'm working with), this does not work because the multi-line values will get mixed up with other variables.
Answering my own question since I couldn't find this elsewhere:
$ env -0 | sort -z | tr '\0' '\n'
env -0 separates each variable by a null character (which is more-or-less how they are already stored internally). sort -z uses null characters instead of newlines as the delimiter for fields to be sorted, and finally tr '\0' '\n' replaces the nulls with newlines again.
Note: env -0 and sort -z are non-standard extensions provided by the GNU coreutils versions of these utilities. Open to other ideas for how to do this with POSIX sort--I'm sure it is possible but it might require a for loop or something; not as easy as a one-liner.
The bash builtin export prints a sorted list of envars:
export -p | sed 's/declare -x //'
Similarly, to print a sorted list of exported functions (without their definitions):
export -f | grep 'declare -fx' | sed 's/declare -fx //'
In a limited environment where env -0 is not available, eg. Alpine 3.13, 3.14 (commands are simplified busybox versions) you can use awk:
awk 'BEGIN { for (K in ENVIRON) { printf "%s=%s%c", K, ENVIRON[K], 0; }}' | sort -z | tr '\0' '\n'
This uses awk to print each environment variable terminated with a null, simulating env -0. Note that setting ORS to null (-vORS='\0') does not work in this limited version of awk, neither does directly printing \0 in the printf, hence the %c to print 0.
Busybox awk lacks any sort functions, hence the remainder of the answer is the same as the top one.
env | sort -f
Worked for me.
The -f option makes sort ignore case, which is what you probably want 99% of the time
I found this thread with two solutions for trimming whitespace: piping to xargs and defining a trim() function:
trim() {
local var="$*"
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
I prefer the second because of one comment:
This is overwhelmingly the ideal solution. Forking one or more external processes merely to trim whitespace from a single string is fundamentally insane – particularly when most shells (including bash) already provide native string munging facilities out-of-the-box.
I am getting, for example, the wifi SSID on macOS by piping to awk (when I get comfortable with regular expressions in bash, I won't fork an awk process), which includes a leading space:
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}'
<some-ssid>
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}' | xargs
<some-ssid>
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}' | trim
$ wifi=$(/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}')
$ trim "$wifi"
<some-ssid>
Why does piping to the trim function fail and giving it an argument work?
It is because your trim() function is expecting a positional argument list to process. The $* is the argument list passed to your function. For the case that you report as not working, you are connecting the read end of a pipe to the function inside which you need to fetch from the standard input file descriptor.
In such a case you need to read from standard input using read command and process the argument list, i.e. as
trim() {
# convert the input received over pipe to a a single string
IFS= read -r var
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
for which you can now do
$ echo " abc " | trim
abc
or using a command substitution syntax to run the command that fetches the string, that you want to pass to trim() with your older definition.
trim "$(/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}')"
In this case, the shell expands the $(..) by running the command inside and replaces it with output of the commands run. So now the function sees trim <args> which it interprets as a positional argument and runs the string replacement functions directly on it.
Working with this code
What is the most elegant way to remove a path from the $PATH variable in Bash?
export PATH=`echo ${PATH} | awk -v RS=: -v ORS=: '/SDE/ {next} {print}'` | sed 's/:*$//'
In this instance if I run just the:
export PATH=`echo ${PATH} | awk -v RS=: -v ORS=: '/SDE/ {next} {print}'`
I can get the path containing /SDE/ removed; however a : remains. The sed command after I am assuming should remove that. When I run this entire command at once nothing gets removed at all. What is wrong with the sed statement that is causing the path not to update and how can I make the sed command remove the colon : after the /SDE/ path variable is removed.
The problem is the placement of the closing back-quote ` in the command:
export PATH=`echo ${PATH} | awk -v RS=: -v ORS=: '/SDE/ {next} {print}'` | sed 's/:*$//'
If you used the recommended $(...) notation, you'd see that this is equivalent to:
export PATH=$(echo ${PATH} | awk -v RS=: -v ORS=: '/SDE/ {next} {print}') | sed 's/:*$//'
which pipes the output of the export operation to sed, but export is silent.
Use:
export PATH=$(echo ${PATH} | awk -v RS=: -v ORS=: '/SDE/ {next} {print}' | sed 's/:*$//')
I have fixed the answer from which the erroneous command was copied verbatim. As tripleee notes in a comment, I'm not wholly convinced by the awk solution, but the question was 'what was wrong with the code' and the answer is 'where the back-quotes are placed'. The awk script does handle removing elements of a PATH at any position in the PATH; the sed script simply ensures there is no trailing : so that there is no implicit use of the current directory at the end of the PATH.
See also: How do I manipulate PATH elements in shell scripts and the clnpath script at How to keep from duplicating PATH variable in csh — the script is for POSIX-ish shells like the Bourne, Korn, Bash shells, despite the question's subject. One difference between clnpath and the notation used here is that clnpath only removes full pathnames; it does not attempt to do partial path element matching:
export PATH=$(clnpath $PATH /opt/SDE/bin)
if the path element to be removed was /opt/SDE/bin. Note that clnpath can be used to maintain LD_LIBRARY_PATH, CDPATH, MANPATH and any other path-like variable; so can the awk invocation, of course.
I note in passing that that the /SDE/ pattern in the awk script will remove /opt/USDER/bin; the slashes in the regex have nothing to do with slashes in the pathname.
$ PATH="$( awk -v rmv="/SDE/" -v path="$PATH" 'BEGIN{sub(rmv,"",path); gsub(/:+/,":",path); print path}' )"
The above is a guess since you didn't provide sample input or expected output so it may not be quite what you want but it is the right approach.
Just in bash:
tmp=":$PATH"
[[ $tmp =~ :[^:]*/SDE/[^:]* ]] && tmp=${tmp/${BASH_REMATCH[0]}/}
PATH=${tmp#:}
In bash
echo ${!X*}
will print all the names of the variables whose name starts with 'X'.
Is it possible to get the same with an arbitrary pattern, e.g. get all the names of the variables whose name contains an 'X' in any position?
Use the builtin command compgen:
compgen -A variable | grep X
This should do it:
env | grep ".*X.*"
Edit: sorry, that looks for X in the value too.
This version only looks for X in the var name
env | awk -F "=" '{print $1}' | grep ".*X.*"
As Paul points out in the comments, if you're looking for local variables too, env needs to be replaced with set:
set | awk -F "=" '{print $1}' | grep ".*X.*"
Easiest might be to do a
printenv |grep D.*=
The only difference is it also prints out the variable's values.
This will search for X only in variable names and output only matching variable names:
set | grep -oP '^\w*X\w*(?==)'
or for easier editing of searched pattern
set | grep -oP '^\w*(?==)' | grep X
or simply (maybe more easy to remember)
set | cut -d= -f1 | grep X
If you want to match X inside variable names, but output in name=value form, then:
set | grep -P '^\w*X\w*(?==)'
and if you want to match X inside variable names, but output only value, then:
set | grep -P '^\w*X\w*(?==)' | grep -oP '(?<==).*'
Enhancing Johannes Schaub - litb answer removing fork/exec in modern bash we could do
compgen -A variable -X '!*X*'
i.e an X in any position in the variable list.
env | awk -F= '{if($1 ~ /X/) print $1}'
To improve on Johannes Schaub - litb's answer:
There is a shortcut for -A variable and a flag to include a pattern:
compgen -v -X '!*SEARCHED*'
-v is a shortcut for -A variable
-X takes a pattern that must not be matched.
Hence -v -X '!*SEARCHED*' reads as:
variables that do not, not match "anything + SEARCHED + anything"
Which is equivalent to:
variables that do match "anything + SEARCHED + anything"
The question explicitly mentions "variables" but I think it's safe to say that many people will be looking for "custom declared things" instead.
But neither functions nor aliases are listed by -v.
If you are looking for variables, functions and aliases, you should use the following instead:
compgen -av -A function -X '!*SEARCHED*'
# equivalent to:
compgen -A alias -A variable -A function -X '!*SEARCHED*'
And if you only search for things that start with a PREFIX, compgen does that for you by default:
compgen -v PREFIX
You may of course adjust the options as needed, and the official doc will help you: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html
to expand Phi's and Johannes Schaub - litb's answers for the following use case:
print contents of all environment variables whose names match a pattern as strings which can be reused in other (Bash) scripts, i.e. with all special characters properly escaped and the whole contents quoted
In case you have the following environment variables
export VAR_WITH_QUOTES=\"FirstName\ LastName\"\ \<firstname.lastname#example.com\>
export VAR_WITH_WHITESPACES="
a bc
"
export VAR_EMPTY=""
export VAR_WITH_QUOTES_2=\"\'
then the following snippet prints all VAR* environment variables in reusable presentation:
for var in $(compgen -A export -X '!VAR*'); do
printf "%s=%s\n" "$var" "${!var#Q}"
done
Snippet is is valid for Bash 4+.
The output is as follows, please note output for newlines, empty variables and variables which contain quotation characters:
VAR_EMPTY=''
VAR_WITH_QUOTES='"FirstName LastName" <firstname.lastname#example.com>'
VAR_WITH_QUOTES_2='"'\'''
VAR_WITH_WHITESPACES=$' \n\ta bc\n'
This also relates to the question Escape a variable for use as content of another script