I want to do the following
export LOGS=//server/log_files/2014_*/server_{1,2,3}
so I can do something like
grep 'Exception' $LOGS/log.txt
I tried alias as well, but I can't get it to not expand.
How could I do this?
Without export, right hand side of an assignment doesn't go through path nor brace expansion.
With export, though, brace expansion is performed. You can prevent it by quoting the value:
export LOGS='//server/log_files/2014_*/server_{1,2,3}'
If you want to use such a value, though, you have to use eval:
eval grep 'Exception' $LOGS/log.txt
You're in the case where you want extended globs. That's the cleanest and the most semantically correct here, as you want to match filenames. As I'm overly pedantic, I'd argue that brace expansions are not the right tool for your task.
# This defines a string that will glob
# No pathname expansions are performed at this step
logs_glob='//server/log_files/2014_*/server_#(1|2|3)'
# You need to activate extended globs with extglob
# To have a failure when no files match the glob, you need failglob
shopt -s failglob extglob
# Unquoted variable $logs_glob, as pathname expansion is desirable
grep 'Exception' $logs_glob
Some will argue that with glob techniques you can't properly handle spaces in names. In fact, you have two ways: either use ? as a wildcard (this will match any character, hence spaces in particular) or use the character class [[:space:]]. This character class will match any space (regular spaces, newlines, tabs, etc.)
Another technique is to use arrays, still with extended globs. I'd argue that this is cleaner.
shopt -s extglob nullglob
# This will populate array with all matching filenames.
# If no matches, array is empty (since we shopted nullglob)
logs_array=( //server/log_files/2014_*/server_#(1|2|3) )
# Before you launch you command with the array, make sure it's not empty:
if ((${#logs_array[#]}!=0)); then
# Observe the quotes for the expansion of the array
grep 'Exception' "${logs_array[#]}"
fi
Related
I wonder whether there is a way to specify an one-or-more modifier for a character class in a parameter substitution pattern in bash, similar to a regex: i.e. [a-z]+. It seems like that for instance to remove all trailing whitespaces from a variable I would need to use ${fn##*[[:space:]]} (would only work for fn=" test" but not for fn=" test a") however I was hoping for something like ${fn##[[:space:]]+}. Is there a special syntax to specify regex instead of the patterns? Where is the pattern format specified?
Using extglob, you can do this:
shopt -s extglob
fn=" test a"
echo "${fn##+([[:space:]])}"
test a
Here glob expression +([[:space:]]) matches 1 or more whitespace characters.
You cannot use regular expressions in parameter expansions like that. However, the extended pattern syntax enabled by extglob is equivalent in power.
$ fn=" test a"
$ echo "$fn"
test a
$ shopt -s extglob
$ echo "${fn##*([[:space:]])}"
test a
How can I match environment variables which include the case-insensitive segment "proxy" that is not a prefix? I'm on bash:
root#PDPINTDEV9:~# echo ${SHELL}
/bin/bash
I want to unset a bunch of proxy variables simultaneously. They all have "proxy" or "PROXY" in the name, such as http_proxy or NO_PROXY. I would like to use glob expansion, which this answer & comment says is what bash uses.
Also based on that answer, I see that I can find environment vars which start with "PROXY":
root#PDPINTDEV9:~# echo "${!PROXY*}"
PROXY_IP PROXY_PORT
But that doesn't make sense with what I've read about glob expansion. Based on those, "${!PROXY*}" should match anything that doesn't start with proxy... I think.
Furthermore, I can't get anything that does make sense with glob syntax to actually work:
root#PDPINTDEV9:~# echo ${*proxy}
-bash: ${*proxy}: bad substitution
root#PDPINTDEV9:~# echo "${!*[pP][rR][oO][xX][yY]}"
-bash: ${!*[pP][rR][oO][xX][yY]}: bad substitution
SOLVED below: Turns out you can't. Crazy, but thanks everyone.
Variable name expansion, as a special case of shell parameter expansion, does not support globbing. But it has two flavors:
${!PREFIX*}
${!PREFIX#}
In both, the * and # characters are hard-coded.
The first form will expand to variable names prefixed with PREFIX and joined by the first character of the IFS (which is a space, by default):
$ printf "%s\n" "${!BASH*}"
BASH BASHOPTS BASHPID BASH_ALIASES BASH_ARGC BASH_ARGV BASH_CMDS BASH_COMMAND ...
The second form will expand to variable names (prefixed with PREFIX), but as separate words:
$ printf "%s\n" "${!BASH#}"
BASH
BASHOPTS
BASHPID
BASH_ALIASES
BASH_ARGC
...
Both of these forms are case-sensitive, so to get the variable names in a case-insensitive manner, you can use set, in combination with some cut and grep:
$ (set -o posix; set) | cut -d= -f1 | grep -i ^proxy
PROXY_IP
proxy_port
But that doesn't make sense with what I've read about glob expansion.
Based on those, "${!PROXY*}" should match anything that doesn't start
with proxy... I think.
No and no.
In the first place, the ! character is not significant to pathname expansion, except when it appears at the beginning of a character class in a pattern, in which case the sense of the class is inverted. For example, fo[!o] is a pattern that matches any three-character string whose first two characters are "fo" and whose third is not another 'o'. But there is no character class in your expression.
But more importantly, pathname expansion isn't relevant to your expression ${!PROXY*} at all. There is no globbing there. The '!' and '*' are fixed parts of the syntax for one of the forms of parameter expansion. That particular expansion produces, by definition, the names of all shell variables whose names start with "PROXY", separated by the first character of the value of the IFS variable. Where it appears outside of double quotes, it is equivalent to ${!PROXY#}, which is less susceptible to globbing-related confusion.
Furthermore, I can't get anything that does make sense with glob syntax to actually work: [...]
No, because, again, there is no globbing going on. You need exactly ${! followed by the name prefix of interest, followed by *} or #} to form the particular kind of parameter expansion you're asking about.
How can I match environment variables which include the case-insensitive segment "proxy"?
You need to explicitly express the case variations of interest to you. For example:
${!PROXY*} ${!proxy*} ${!Proxy*}
I am trying the following command:
ls myfile.h1.{`seq -s ',' 3501 3511`}*
But ls raises the error:
ls: cannot access myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511}*: No such file or directory
Seems like ls is thinking the entire line is a filename and not a wildcard pattern. But if I just copy that command ls myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511}* in the terminal I get the listing as expected.
Why does typing out the command in full work, but not the usage with seq?
seq is not needed for your case, try
$ ls myfile.h1.{3500..3511}
if you want to use seq I would suggest using format option
$ ls $(seq -f 'myfile.h1.%g' 3501 3511)
but I don't think there is any reason to do so.
UPDATE:
Note that I didn't notice the globbing in the original post. With that, the brace extension still preferred way
$ ls myfile.h1.{3500..3511}*
perhaps even factoring the common digit out, if your bash support zero padding
$ ls myfile.h1.35{00..11}*
if not you can extract at least 3 out
$ ls myfile.h1.3{500..511}*
Note that the seq alternative won't work with globbing.
Other answer has more details...
karakfa's answer, which uses a literal sequence brace expansion expression, is the right solution.
As for why your approach didn't work:
Bash's brace expansion {...} only works with literal expressions - neither variable references nor, as in your case, command substitutions (`...`, or, preferably, $(...)) work[1] - for a concise overview, see this answer of mine.
With careful use of eval, however, you can work around this limitation; to wit:
from=3501 to=3511
# CAVEAT: Only do this if you TRUST that $from and $to contain
# decimal numbers only.
eval ls "myfile.h1.{$from..$to}*"
#ghoti suggests the following improvement in a comment to make the use of eval safe here:
# Use parameter expansion to remove all non-digit characters from the values
# of $from and $to, thus ensuring that they either contain only a decimal
# number or the empty string; this expansion happens *before* eval is invoked.
eval ls "myfile.h1.{${from//[^0-9]/}..${to//[^0-9]/}}*"
As for how your command was actually evaluated:
Note: Bash applies 7-8 kinds of expansions to a command line; only the ones that actually come into play here are discussed below.
first, the command in command substitution `seq -s ',' 3501 3511` is executed, and replaced by its output (also note the trailing ,):
3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,
the result then forms a single word with its prefix, myfile.h1.{ and its suffix, }*, yielding:
myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,}*
pathname expansion (globbing) is then applied to the result - in your case, since no files match, it is left as-is (by default; shell options shopt -s nullglob or shopt -s failglob could change that).
finally, literal myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,}* is passed to ls, which - because it doesn't refer to an existing filesystem item - results in the error message you saw.
[1] Note that the limitation only applies to sequence brace expansions (e.g., {1..3}); list brace expansions (e.g, {1,2,3}) are not affected, because no up-front interpretation (interpolation) is needed; e.g. {$HOME,$USER} works, because brace expansion results expanding the list to separate words $HOME, and $USER, which are only later expanded.
Historically, sequence brace expansions were introduced later, at a time when the order of shell expansions was already fixed.
If I want to match a file called "-f" or "-r" I might do something like
test.sh -?
And if I want to send the literal '-?' to a program as an argument I might do something like:
test.sh -\?
If no such file "-f" or "-r" or anything like it exists, then what should my shell do with
test.sh -?
Should it tell me that no file matches this pattern?
In bash, the default is to treat an unmatched pattern literally. If the nullglob option is set, an unmatched pattern "evaporates"; it is removed from command, not even expanding to the empty string.
In zsh, an unmatched pattern produces an error by default. Setting the nomatch option causes an unmatched pattern to be treated literally, and zsh also supports a nullglob option which causes unmatched patterns to disappear. There is also a cshnullglob option which acts like nullglob, but requires at least one pattern in a command to match, or an error is produced.
Note that POSIX specifies that if the pattern contains an invalid bracket expression or does not match any existing filenames or pathnames, the pattern string shall be left unchanged in sh.
ash, dash, ksh, bash and zsh all behave this way when invoked as sh.
POSIX specifies that if the pattern contains an invalid bracket expression or does not match any existing filenames or pathnames, the pattern string shall be left unchanged in sh.
ash, dash, ksh, bash and zsh all behave this way when invoked as sh.
You seem to be looking for the nullglob option, at least with Bash:
shopt -s nullglob
Without the nullglob option, an unmatched pattern is passed as its literal self to your program: the shell will pass -? to the script if there isn't a file that matches. With the nullglob option, unmatched patterns are replaced with nothing at all.
If no such pattern exists, the shell, by default, just returns the pattern your gave, including whatever * or ? characters you used. To determine whether the file actually exists, test it. Thus, inside your script, use:
[ -f "$1" ] || echo "no such file exists"
I'd like list files with the name pattern like [max|min].txt, so execute
ls [max|min].txt in bash shell, but it doesn't work, and the error message I got is:
ls: cannot access [max: No such file or directory
so what's the right way to do this job?
Square brackets are for character matching, and vertical bars are for pipes. You're looking for brace expansion.
ls {max,min}.txt
Bash has a shell option called extglob that you can enable with the command shopt -s extglob. This will allow you to use the pattern format #(pattern-list) where pattern-list is a pipe separated list of patterns. It will match against filenames and will exclude any pattern that does not match a filename, just like the [abc] range expression. Bash also has brace expansion, but this does not appear to be what you are asking for, as brace expansion does not match against filenames or expand like wildcards or range expressions do.
$ shopt -s extglob
$ touch max.txt min.txt
$ echo #(max|min).txt
max.txt min.txt
$ echo #(min|mid|max).txt
max.txt min.txt
$ echo {min,mid,max}.txt
min.txt mid.txt max.txt
A couple of things to note about the sequence of commands above:
echo #(mid|min|max).txt does not output mid.txt because there is no file that matches.
echo #(min|mid|max).txt re-orders the output to be sorted, in the same manner as a wildcard expansion.
echo {min,mid,max}.txt is brace expansion and outputs all elements in the order given.