How can I match environment variables which include the case-insensitive segment "proxy" that is not a prefix? I'm on bash:
root#PDPINTDEV9:~# echo ${SHELL}
/bin/bash
I want to unset a bunch of proxy variables simultaneously. They all have "proxy" or "PROXY" in the name, such as http_proxy or NO_PROXY. I would like to use glob expansion, which this answer & comment says is what bash uses.
Also based on that answer, I see that I can find environment vars which start with "PROXY":
root#PDPINTDEV9:~# echo "${!PROXY*}"
PROXY_IP PROXY_PORT
But that doesn't make sense with what I've read about glob expansion. Based on those, "${!PROXY*}" should match anything that doesn't start with proxy... I think.
Furthermore, I can't get anything that does make sense with glob syntax to actually work:
root#PDPINTDEV9:~# echo ${*proxy}
-bash: ${*proxy}: bad substitution
root#PDPINTDEV9:~# echo "${!*[pP][rR][oO][xX][yY]}"
-bash: ${!*[pP][rR][oO][xX][yY]}: bad substitution
SOLVED below: Turns out you can't. Crazy, but thanks everyone.
Variable name expansion, as a special case of shell parameter expansion, does not support globbing. But it has two flavors:
${!PREFIX*}
${!PREFIX#}
In both, the * and # characters are hard-coded.
The first form will expand to variable names prefixed with PREFIX and joined by the first character of the IFS (which is a space, by default):
$ printf "%s\n" "${!BASH*}"
BASH BASHOPTS BASHPID BASH_ALIASES BASH_ARGC BASH_ARGV BASH_CMDS BASH_COMMAND ...
The second form will expand to variable names (prefixed with PREFIX), but as separate words:
$ printf "%s\n" "${!BASH#}"
BASH
BASHOPTS
BASHPID
BASH_ALIASES
BASH_ARGC
...
Both of these forms are case-sensitive, so to get the variable names in a case-insensitive manner, you can use set, in combination with some cut and grep:
$ (set -o posix; set) | cut -d= -f1 | grep -i ^proxy
PROXY_IP
proxy_port
But that doesn't make sense with what I've read about glob expansion.
Based on those, "${!PROXY*}" should match anything that doesn't start
with proxy... I think.
No and no.
In the first place, the ! character is not significant to pathname expansion, except when it appears at the beginning of a character class in a pattern, in which case the sense of the class is inverted. For example, fo[!o] is a pattern that matches any three-character string whose first two characters are "fo" and whose third is not another 'o'. But there is no character class in your expression.
But more importantly, pathname expansion isn't relevant to your expression ${!PROXY*} at all. There is no globbing there. The '!' and '*' are fixed parts of the syntax for one of the forms of parameter expansion. That particular expansion produces, by definition, the names of all shell variables whose names start with "PROXY", separated by the first character of the value of the IFS variable. Where it appears outside of double quotes, it is equivalent to ${!PROXY#}, which is less susceptible to globbing-related confusion.
Furthermore, I can't get anything that does make sense with glob syntax to actually work: [...]
No, because, again, there is no globbing going on. You need exactly ${! followed by the name prefix of interest, followed by *} or #} to form the particular kind of parameter expansion you're asking about.
How can I match environment variables which include the case-insensitive segment "proxy"?
You need to explicitly express the case variations of interest to you. For example:
${!PROXY*} ${!proxy*} ${!Proxy*}
Related
I have two text files identical to each other a.text and b.text with the same content.
Content of a.text and b.text
abcd
target
efgh
Can anyone explain why one of the commands work but not the other and if there is a way of making it work?
Output of command 1
grep "target" {a,b}.text
>>a.text:target
b.text:target
Output of command 2
file="{a,b}.text"
grep "target" $file
>>grep: {a,b}.text: No such file or directory
Happy if someone can point me to a location where I can read more about this as well. I can only assume that when storing it as a variable it explicitly looks for a file called {a,b}.text although, what I am not as sure about is why and what leads to that.
As user1934428 said, brace expansion happens before parameter expansion. Quoting from the bash manual:
Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
To get this to work you can store the file names in an array. Arrays can hold multiple file names without being subject to quoting or expansion issues that plague normal string variables.
files=({a,b}.text)
grep "target" "${files[#]}"
This works because {a,b} is now evaluated when the variable is assigned, rather than when it is expanded.
In the first case, you have a brace-expansion. In the second case, you are searching a file with the literal name {a,b}.text. The reason is that in bash, brace expansion happens before parameter expansion.
This question already has answers here:
Stripping prefixes and suffixes from shell words matching a pattern
(2 answers)
Difference between ${} and $() in Bash [duplicate]
(3 answers)
Closed 1 year ago.
I have a string with the structure task_name-student_name and I want to split it into two variables:
task: containing the chunk before the -
student: containing the chunk after the -
I can get this to work with sed:
string="task_name-student_name"
student=$(echo "$string" | sed "s/.*-//")
task=$(echo "$string" | sed "s/-[^-]*$//")
However, VS Code suggests "See if you can use $(variable//search/replace) instead".
So I have two questions:
Why would $(variable//search/replace) be better
How do I get the parameter expansion to work without it being interpreted as a command?
When I try
echo $("$string"//-[^-]*$//)
or
echo $(echo $("$string"//-[^-]*$//))
I get this output:
bash: task_name-student_name//-[^-]*$//: No such file or directory
Thanks in advance!
First: for variable expansion, you want curly braces instead of parentheses. $(something) will execute something as a command; ${something} will expand something as a variable. And just for completeness, $((something)) will evaluate something as an arithmetic expression (integers only, no floating point).
As for replacing the sed with a variable expansion: I wouldn't use $(variable//search/replace} for this; there are more appropriate modifications. ${variable#pattern} will remove the shortest possible match of pattern from the beginning of the variable's value, so use that with the pattern *- to remove through the first "-":
student=${string#*-}
Similarly, ${variable%pattern} will remove from the end of the variable's value, so you can use this with the pattern -* to remove from the dash to the end:
task=${string%-*}
Note that the patterns used here are "glob" expressions (like filename wildcards), not regular expressions like sed uses; they're just different enough to be confusing. Also, the way I've written these assumes there's exactly one "-" in the string; if there's a possibility some student will have a hyphenated name or something like that, you may need to modify them.
There are lots more modifications you can do in a parameter expansion; see the bash hacker's wiki on the subject. Some of these modifications will work in other shells besides bash; the # and % modifiers (and the "greedy" versions, ## and %%) will work in any shell that conforms to the POSIX standard.
I know that when you use a for loop in Bash, the items that you loop through are separated using the $IFS variable.
However, if I run the following commands, I correctly show the two files I have created - even though they have spaces:
touch file\ {1..2}.txt
for file in *.txt; do
echo "Found: ${file}"
done
The output is:
Found: file 1.txt
Found: file 2.txt
I am assuming that this is because when the shell sees the wildcard pattern, it expands it and escapes any special characters or whitespace. This is in contrast to if I run:
touch file\ {1..2}.txt
files=$(ls *.txt)
for file in $files; do
echo "Found: ${file}"
done
This results in:
Found: file
Found: 1.txt
Found: file
Found: 2.txt
Which makes sense - by default $IFS contains whitespace, so the file names are split.
What I want to understand is:
Am I correct that wildcard expansion results in a set of strings that contain escaped special characters
Where is it documented that this is the case, if I am correct?
Is there any way to show that this is happening?
I was hoping I could use something like set -x to show what the wildcard expands to and actually see the escaped characters, because I really want to be able to understand what is going on here.
I am writing a long series of articles on effective shell usage (effective-shell.com) and I'm struggling to find a way to explain the differences of behaviour here, I'm assuming that the shell is escaping characters but I'd like to know if this is the case and how to see it if possible!
Thanks in advance.
done
Am I correct that wildcard expansion results in a set of strings that contain escaped special characters
No. There is no need for the shell to escape special characters at that point, because filename expansion is the last word expansion to be performed; strings resulting from it are not subjected to word splitting or any other expansion; they stay as-is. This is documented in the manual as follows:
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion.
I seem to be able to create environment variables that execute commands; like this:
$ cat ./src
FOO="echo"
$ . ./src
$ echo $FOO
echo
$ $FOO hello
hello
$
Is there a way I can modify that environment variable so that it prefixes the setting of another environment variable before the command? I.e. is there a way to work around the following problem?
$ cat ./src
FOO="MY_DIR=/tmp echo"
$ . ./src
$ echo $FOO
MY_DIR=/tmp echo
$ $FOO hello
-bash: MY_DIR=/tmp: No such file or directory
$
I.e. what I'd like to happen is to have an environment variable that does the equivalent of the following manually typed in the shell:
$ MY_DIR=/tmp echo hello
hello
$
...similar to how sans envvar-prefix, $FOO effectively had the same effect as typing echo at the shell.
/tmp/ exists of course, btw:
$ ls -ld /tmp/
drwxrwxrwt. 25 root root 500 May 19 11:35 /tmp/
$
Update:
I have a constraint that "FOO" must be invoked like $FOO hello and not FOO hello. So unfortunately a function like in #John Kugelman's (current) answer can't be a solution, even if it's more proper.
It's best to put data into variables, code into functions. Functions are more natural, expressive, and flexible than variables holding code. They look just like any other command but can take arbitrary actions, including but not limited to prepending commands and variable assignments.
foo() {
MY_DIR=/tmp echo "$#"
}
foo hello
Here "$#" is a placeholder for the arguments passed to foo().
I have a constraint that "FOO" must be invoked like $FOO hello and not FOO hello.
That constraint is impossible, I'm afraid.
I am curious about the mechanics of what's going on here: i.e. why can you make an environment variable that's sort of "aliased" to a command (I know true aliasing is something else), but that mechanism doesn't accommodate the seemingly small change to prefix "stuff" to the command?
Bash expands commands in several passes in a fixed, prescribed order. Very early on it splits the command into words and then marks the variable assignments with invisible flags. It expands $variable references in a later pass. It doesn't look at the results to see if they look like additional variable expansions. The equal signs are effectively ignored.
If you want to know the nitty gritty details, open up the Bash man page. It's incredibly long and the details are scattered throughout. Let me pull out the key sections and some choice quotes to help you digest it:
Shell Grammar, Simple Commands
A simple command is a sequence of optional variable assignments followed by blank-separated words and redirections, and terminated by a control operator.
Simple Command Expansion
When a simple command is executed, the shell performs the following expansions, assignments, and redirections, from left to right.
The words that the parser has marked as variable assignments (those preceding the command name) and redirections are saved for later processing.
The words that are not variable assignments or redirections are expanded. If any words remain after expansion, the first word is taken to be the name of the command and the remaining words are the arguments.
...
If no command name results, the variable assignments affect the current shell environment. Otherwise, the variables are added to the environment of the executed command and do not affect the current shell environment.
Expansion
Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
Expansion, Parameter Expansion
The $ character introduces parameter expansion, command substitution, or arithmetic expansion.
Assignments are marked in step 1 and variables (AKA parameters) are expanded in step 4.
The only things that happen after variable expansion are:
Word splitting. A variable can expand to multiple words if it contains whitespace. (Or to be more precise, if it contains any of the characters in the inter-field separator variable $IFS.)
Pathname expansion. Also known as globbing, or wildcards. If a variable contains *, ?, or [ they'll be expanded to the names of matching files, if there are any.
Quote removal. This pass happens after variable expansion, but it specifically does not apply to the results of any previous expansion step. So quotes the user typed are removed, but quotes that were the results of a substitution are retained.
Neither word splitting nor pathname expansion are what you need, so that's why it's not possible to store an assignment in a variable.
Suppose you have something like:
$ a=(fooa foob foox)
Then you can do:
$ b=(${(M)a:#*(a|b)})
To select a's elements matching the pattern.
So you have:
$ print ${(qq)b}
'fooa' 'foob'
Then you expect to build the pattern in some dynamic way, so you have it in another variable, say:
$ p="*(a|b)"
And you expect this:
$ b=(${(M)a:#$p})
Would work the same as before, as the documentation says, but it doesn't:
$ print ${(qq)b}
''
Why is that?
Because zsh tries to select $p's value literally (a plain string text) in this case:
a=('*(a|b)' fooa foob)
p="*(a|b)"
b=(${(M)a:#$p})
print ${(qq)b}
;#⇒'*(a|b)'
We could tell zsh to treat $p's expansion as patterns rather than literal values explicitly by a ${~spec} form.
${~spec}
Turn on the GLOB_SUBST option for the evaluation of spec; if the ‘~’ is doubled, turn it off. When this option is set, the string resulting from the expansion will be interpreted as a pattern anywhere that is possible, such as in filename expansion and filename generation and pattern-matching contexts like the right hand side of the ‘=’ and ‘!=’ operators in conditions.
-- zshexpn(1): Expansion, Parameter Expansion
In this case, we could use it like this:
a=(fooa foob foox)
p="*(a|b)"
b=(${(M)a:#${~p}}) ;# tell zsh treat as a pattern for `$p`
print ${(qq)b}
;#⇒'fooa' 'foob'
Note: It gives some hints in the parameter expansion flag b for storing patterns in variable values:
b
Quote with backslashes only characters that are special to pattern matching. This is useful when the contents of the variable are to be tested using GLOB_SUBST, including the ${~...} switch.
Quoting using one of the q family of flags does not work for this purpose since quotes are not stripped from non-pattern characters by GLOB_SUBST. In other words,
pattern=${(q)str}
[[ $str = ${~pattern} ]]
works if $str is ‘a*b’ but not if it is ‘a b’, whereas
pattern=${(b)str}
[[ $str = ${~pattern} ]]
is always true for any possible value of $str.
-- zshexpn(1): Expansion, Parameter Expansion, Parameter Expansion Flags