Get list of variables whose name matches a certain pattern - bash

In bash
echo ${!X*}
will print all the names of the variables whose name starts with 'X'.
Is it possible to get the same with an arbitrary pattern, e.g. get all the names of the variables whose name contains an 'X' in any position?

Use the builtin command compgen:
compgen -A variable | grep X

This should do it:
env | grep ".*X.*"
Edit: sorry, that looks for X in the value too.
This version only looks for X in the var name
env | awk -F "=" '{print $1}' | grep ".*X.*"
As Paul points out in the comments, if you're looking for local variables too, env needs to be replaced with set:
set | awk -F "=" '{print $1}' | grep ".*X.*"

Easiest might be to do a
printenv |grep D.*=
The only difference is it also prints out the variable's values.

This will search for X only in variable names and output only matching variable names:
set | grep -oP '^\w*X\w*(?==)'
or for easier editing of searched pattern
set | grep -oP '^\w*(?==)' | grep X
or simply (maybe more easy to remember)
set | cut -d= -f1 | grep X
If you want to match X inside variable names, but output in name=value form, then:
set | grep -P '^\w*X\w*(?==)'
and if you want to match X inside variable names, but output only value, then:
set | grep -P '^\w*X\w*(?==)' | grep -oP '(?<==).*'

Enhancing Johannes Schaub - litb answer removing fork/exec in modern bash we could do
compgen -A variable -X '!*X*'
i.e an X in any position in the variable list.

env | awk -F= '{if($1 ~ /X/) print $1}'

To improve on Johannes Schaub - litb's answer:
There is a shortcut for -A variable and a flag to include a pattern:
compgen -v -X '!*SEARCHED*'
-v is a shortcut for -A variable
-X takes a pattern that must not be matched.
Hence -v -X '!*SEARCHED*' reads as:
variables that do not, not match "anything + SEARCHED + anything"
Which is equivalent to:
variables that do match "anything + SEARCHED + anything"
The question explicitly mentions "variables" but I think it's safe to say that many people will be looking for "custom declared things" instead.
But neither functions nor aliases are listed by -v.
If you are looking for variables, functions and aliases, you should use the following instead:
compgen -av -A function -X '!*SEARCHED*'
# equivalent to:
compgen -A alias -A variable -A function -X '!*SEARCHED*'
And if you only search for things that start with a PREFIX, compgen does that for you by default:
compgen -v PREFIX
You may of course adjust the options as needed, and the official doc will help you: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html

to expand Phi's and Johannes Schaub - litb's answers for the following use case:
print contents of all environment variables whose names match a pattern as strings which can be reused in other (Bash) scripts, i.e. with all special characters properly escaped and the whole contents quoted
In case you have the following environment variables
export VAR_WITH_QUOTES=\"FirstName\ LastName\"\ \<firstname.lastname#example.com\>
export VAR_WITH_WHITESPACES="
a bc
"
export VAR_EMPTY=""
export VAR_WITH_QUOTES_2=\"\'
then the following snippet prints all VAR* environment variables in reusable presentation:
for var in $(compgen -A export -X '!VAR*'); do
printf "%s=%s\n" "$var" "${!var#Q}"
done
Snippet is is valid for Bash 4+.
The output is as follows, please note output for newlines, empty variables and variables which contain quotation characters:
VAR_EMPTY=''
VAR_WITH_QUOTES='"FirstName LastName" <firstname.lastname#example.com>'
VAR_WITH_QUOTES_2='"'\'''
VAR_WITH_WHITESPACES=$' \n\ta bc\n'
This also relates to the question Escape a variable for use as content of another script

Related

Bash command to parse .env file contains equal (=) sign in values

I have a .env file,
# Some variables
USERNAME=user1
PASSWORD=
# Other variables
URL=https://example.com
TOKEN=muDm8qJ6mqM__YX024dk1RXluPnwd-3mxyt1LwLoI4ISiVPA==
Purpose
My purpose is to create one line bash command to print each non-empty variables in <name> <value> format, which shall
Print one line for each variable with [space] between name and
value;
Ignore comments;
Ignore variables with no value;
Expected output
The expected output is,
USERNAME user1
URL https://example.com
TOKEN muDm8qJ6mqM__YX024dk1RXluPnwd-3mxyt1LwLoI4ISiVPA==
Current Solution
My current solution is as following,
grep -v '^#' .env | grep -v '^[[:space:]]*$' | grep -v '=$' | sed 's/=/ /' | while read -r line; do echo $line; done
Actual output
USERNAME user1
URL https://example.com
It only prints out the first two lines, the issue with last line is caused by the equal (=) signs in TOKEN value.
Help needed
Anyone can help me to rectify the command? also welcome if there is easier way to achieve the goal.
Using sed
$ sed '/=\</!d;s/=/ /' input_file
USERNAME user1
URL https://example.com
TOKEN muDm8qJ6mqM__YX024dk1RXluPnwd-3mxyt1LwLoI4ISiVPA==
Delete lines that do not match a valid variable i.e have no inpurt after =, this will also match blank lines and for your example data, lines with comments as there is no valid variable on the line. Finally, replace the first occurance of the = equal character for a space.
The only way to process any environment variable file is to actually interpret it. Various methods for printing out the variable lists are described below.
Various “environment files” are usually designed to be run by POSIX shell to evaluate all the variables. That makes it possible to call external commands, perform calculations etc. when assigning to the variables.
Do this only is you trust you input file. Any command can be executed by parsing the file using the script below.
You can source the file (builtin command .):
. .env
After you source it, you can evaluate the variables. If you know what all variables you want to print, use this (for vars USERNAME and PASSWORD)
. .env
for var_name in USERNAME PASSWORD
do
printf "%s %s\n" "$var_name" "${!var_name}"
done
If you want to print all the variables that are explicitly specified, use this approach:
. .env
grep '^\w+=' | sed 's/=//' | while read var_name
do
printf "%s %s\n" "$var_name" "${!var_name}"
done
But this solution is also not perfect, because the .env file can use constructions different from var_name=value to assign the vars. Use the set command to print all variables.
This prints out absolutely all variables. (In reality, the environment file should inherit its variables from the system environment vars, so there is a lot of them.)
. .env
printenv | sed 's/=/ /'
The sed is used to get rid of the equals signs.
Beware of some dangerous variable values. The variables can contain even newline character, backspaces and other control characters.
There is also a lot of various utilities for printing the variable in a way that allow loading them back to the shell. For example:
declare -p or typeset -p — prints out declare commands to declare all the currently set shell variables
export -p — the same, but for environment variables (that is probably that thing you want)
set — prints out all shell variables
If the file contains only comments and simple key=value lines without # chars in strings, you can use this pipeline:
sed 's/#.*//' | fgrep = | sed 's/=/ /'
Or if you do not want to output empty variables, use this:
sed 's/#.*//' | grep =. | sed 's/=/ /'

print environment variables sorted by name including variables with newlines

I couldn't find an existing answer to this specific case: I would like to simply display all exported environment variables sorted by their name. Normally I can do this like simply like:
$ env | sort
However, if some environment variables contain newlines in their values (as is the case on the CI system I'm working with), this does not work because the multi-line values will get mixed up with other variables.
Answering my own question since I couldn't find this elsewhere:
$ env -0 | sort -z | tr '\0' '\n'
env -0 separates each variable by a null character (which is more-or-less how they are already stored internally). sort -z uses null characters instead of newlines as the delimiter for fields to be sorted, and finally tr '\0' '\n' replaces the nulls with newlines again.
Note: env -0 and sort -z are non-standard extensions provided by the GNU coreutils versions of these utilities. Open to other ideas for how to do this with POSIX sort--I'm sure it is possible but it might require a for loop or something; not as easy as a one-liner.
The bash builtin export prints a sorted list of envars:
export -p | sed 's/declare -x //'
Similarly, to print a sorted list of exported functions (without their definitions):
export -f | grep 'declare -fx' | sed 's/declare -fx //'
In a limited environment where env -0 is not available, eg. Alpine 3.13, 3.14 (commands are simplified busybox versions) you can use awk:
awk 'BEGIN { for (K in ENVIRON) { printf "%s=%s%c", K, ENVIRON[K], 0; }}' | sort -z | tr '\0' '\n'
This uses awk to print each environment variable terminated with a null, simulating env -0. Note that setting ORS to null (-vORS='\0') does not work in this limited version of awk, neither does directly printing \0 in the printf, hence the %c to print 0.
Busybox awk lacks any sort functions, hence the remainder of the answer is the same as the top one.
env | sort -f
Worked for me.
The -f option makes sort ignore case, which is what you probably want 99% of the time

How can I use grep as cat

I am creating a script that parses some files and greps out the necessary information.
I have set up many different variables in arrays that search for different aspects in the files.
e.g. dates, locations, types.
However I wanted to make each of these variables optional which is where I run into an issue.
the syntax of the command would have been simple
grep ${dates} filename | grep ${locations} | grep ${types}
However, due to variables being optional, the above won't work if a variable is unset.
I was trying to find a way to get grep to find anything (i.e. like egrep .* filename)
that way I could set the variables to the proper regex and have the command still run.
unfortunately, when I set the variable to equal '' it freezes, when I set it to '.' it just greps everything from every file in the current directory and when I leave the variable blank it takes the filename as the variable and waits for a filename.
is there anyway that I can set a variable so that grep $variable file outputs the same as cat would?
Many thanks in advance!
To get grep to act like cat use an empty string as a search pattern, i.e. grep "". Therefore to make some of those variables optional, but not have piped greps fail, just quote the variables:
grep "${dates}" filename | grep "${locations}" | grep "${types}"
Demonstration. Search {250,255,260...280} for the digits 5, 2, and 7:
x=5 y=2 z=7 ; seq 250 5 280 | grep "$x" | grep "$y" | grep "$z"
275
Now unset two of the variables, and it still works:
unset x y ; seq 250 5 280 | grep "$x" | grep "$y" | grep "$z"
270
275
If there aren't any $dates in filename then there is nothing to feed the rest of the pipeline.
I think the best way to do it is to grep for each string separately.
If you get a match, then grep for the next string.
Can you post the source file and a target output. From your question it sounds like you just need to use grep -E in conjunction with the | pipe delimiter.
grep -E "${dates}|${locations}|${types}" fileName
The above line should automatically get you every occurrence of the any of the patterns. This is not even a regex yet.

Reference to a bash variable whose name contains dot

I have a bash variable: agent1.ip with 192.168.100.137 as its value. When I refer to it in echo like this:
echo $agent1.ip
the result is:
.ip
How can I access the value?
UPDATE: my variables are:
Bash itself doesn't understand variable names with dots in them, but that doesn't mean you can't have such a variable in your environment. Here's an example of how to set it and get it all in one:
env 'agent1.ip=192.168.100.137' bash -c 'env | grep ^agent1\\.ip= | cut -d= -f2-'
Since bash.ip is not a valid identifier in bash, the environment string bash.ip=192.168.100.37 is not used to create a shell variable on shell startup.
I would use awk, a standard tool, to extract the value from the environment.
bash_ip=$(awk 'BEGIN {print ENVIRON["bash.ip"]}')
The cleanest solution is:
echo path.data | awk '{print ENVIRON[$1]}'
Try this:
export myval=`env | grep agent1.port | awk -F'=' '{print $2}'`;echo $myval
Is your code nested, and using functions or scripts that use ksh?
Dotted variable names are an advanced feature in ksh93. A simple case is
$ a=1
$ a.b=123
$ echo ${a.b}
123
$ echo $a
1
If you first attempt to assign to a.b, you'll get
-ksh: a.b=123: no parent
IHTH

List all environment variable names in busybox

Environment variables with multiline values may confuse env's output:
# export A="B
> C=D
> E=F"
# env
A=B
C=D
E=F
TERM=xterm
SHELL=/bin/bash
USER=root
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/root
LANG=en_US.UTF-8
PS1=\h:\w\$
SHLVL=1
HOME=/root
LOGNAME=root
_=/usr/bin/env
In this case I can not just use awk -F= to extract all names because it shows wrong names C and E:
# env | awk -F= '{print $1}'
A
C
E
TERM
SHELL
USER
MAIL
PATH
PWD
LANG
PS1
SHLVL
HOME
LOGNAME
_
Then I figured out that env supports flag -0 to end each output line with 0 byte rather than newline, so using sed I could cut off the values in bash:
# env -0 | sed -e ':a;N;$!ba;s/\([^=]\+\)=\([^\x00]*\)\x00/\1\n/g'
A
TERM
SHELL
USER
MAIL
PATH
PWD
LANG
PS1
SHLVL
HOME
LOGNAME
_
But BusyBox's version of env does not support flag -0. Is there another way to do it?
If you are using linux (I thought busybox ran only on linux, but I may be wrong), /proc/self/environ contains a NUL separated environment in the same form as env -0 gives you. You can replace env -0 | with < /proc/self/environ.
sed -e ':a;N;$!ba;s/\([^=]\+\)=\([^\x00]*\)\x00/\1\n/g' < /proc/self/environ
This is maybe not an elegant but working solution. It first extracts all possible names from env's output, then verifies each of them using shell's expansion ${parameter:+word}. And finally it removes duplicates, since the same variable name could be printed on several lines in env's output (as a real variable name and as a part of some other variable's multiline value):
env | awk -F= '/[a-zA-Z_][a-zA-Z_0-9]*=/ {
if (!system("[ -n \"${" $1 "+y}\" ]")) print $1 }' | sort | uniq
PS: The | sort | uniq part can be also implemented in awk.
This will break if your environment variable values contain nulls. But that would also break from POSIX compatibility.
So it should work.
...unless you expect to encounter environment variable names which contain newlines. In that case the newlines will be truncated when they're displayed. However I can't seem to fathom how to create an environment variable with a newline in it in a busybox shell. My local shells balk at it at any rate. So I don't think that would be a big problem. As far as POSIX says, Other characters may be permitted by an implementation; applications shall tolerate the presence of such names. so I think stripping them and not erroring-out is tolerable.
# Read our environment; it's delimited by null bytes.
# Remove newlines
# Replace null bytes with newlines
# On each line, grab everything before the first '='
cat /proc/self/environ | tr -d '\n' | tr '\0' '\n' | cut -d '=' -f 1

Resources