When I the following command in bash, I get a list of files that match the regular expression I want:
$> ls *-[0-9].jtl
benchmark-1422478133-1.jtl benchmark-1422502883-4.jtl benchmark-1422915207-2.jtl
However, when I run the same command in the fish shell, I get different result:
$> ls *-[0-9].jtl
fish: No matches for wildcard '*-[0-9].jtl'.
ls *-[0-9].jtl
^
How come?
Fish's documentation does not claim to support the full power of POSIX glob patterns.
Quoting the docs:
Wildcards
If a star (*) or a question mark (?) is present in the parameter, fish attempts to match the given parameter to any files in such a way that:
? can match any single character except /.
* can match any string of characters not containing /. This includes matching an empty string.
** matches any string of characters. This includes matching an empty string. The string may include the / character but does not need to.
Notably, there's no mention of character classes, as fish doesn't support them.
If you want globs guaranteed to support all POSIX (fnmatch) features, use a POSIX-compliant or POSIX-superset shell.
You can also use more extended tool unix find. It is very powerful.
https://kb.iu.edu/d/admm
https://duckduckgo.com/?q=unix+find
example: use regular expressions
find . -path '.*-[0-9].jtl' -not -path '.*-32.jtl'
Fish just needs quotes "*.conf" to do the same as bash *.conf.
This is an older post, but I think it's worth revisiting this. At time of writing (Mar 2021), the documentation does explicitly state supporting wildcards.
Fish supports the familiar wildcard *. To list all JPEG files:
> ls *.jpg
lena.jpg
meena.jpg
santa maria.jpg
You can include multiple wildcards:
> ls l*.p*
lena.png
lesson.pdf
Especially powerful is the recursive wildcard ** which searches directories recursively:
> ls /var/**.log
/var/log/system.log
/var/run/sntp.log
However, I still all too frequently run into this same issue
[/home/glass ]
><glass#rockpiX-Ubuntu> rm *.log.old
fish: No matches for wildcard “*.log.old”. See `help expand`.
rm *.log.old
^
In fish 3+ you could string match:
ls | string match -r --entire '-[0-9].jtl'
options:
-r: regular expression
--entire: returns the entire matching string
Related
I want to list all .jpg files in all subdirectories using ls.
For the same directory this works fine:
ls *.jpg
However, when using the -R for recursiveness:
ls -R *.jpg
I get:
zsh:no matches found: *.jpg
Why does this not work?
Note: I know it can be done using find or grep but I want to know why the above does not work.
The program ls is not designed to handle patterns by itself.
When you run ls -R *.jpg, the pattern *.jpg is not directly passed to ls. The shell replaces it by a list of all files that match the pattern. (Only if there is no file with a matching name, ls will see the file name *.jpg and not find a file of this name.)
Since you are using zsh (with the default setting setopt nomatch), it prints an error message instead of passing the pattern to ls.
If there are matching files, e.g. A.jpg, B.jpg, C.jpg, the command
ls *.jpg
will be run by the shell as
ls A.jpg B.jpg C.jpg
In contrast to this, find is designed to handle patterns with its -name test. When using find you should make sure the pattern is not replaced by the shell, e.g. by using -name '*.jpg' or -name \*.jpg. Otherwise you might get unexpected results or an error if there are matching files in the current directory.
Edit:
As shown in Martin Tournoij's answer you could use the recursive glob pattern ls **/*.jpg, but this is also handled by the shell not by ls, so you don't need option -R. In zsh this recursive pattern ** is enabled by default, in bash you need to enable it with shopt -s globstar.
The shell first expands any glob patterns, and then runs the command. So from ls's perspective, ls *.jpg is exactly the same as if you had typed ls one.jpg two.jpg. The -R flag to ls only makes sense if you use it on a directory, which you're not doing here.
This is also why mv *.jpg *.png doesn't work as expected on Unix systems, since mv never sees those patterns but just the filenames it expanded to (it does on e.g. Windows, where the globbing is done by the program rather than the shell; there are advantages and disadvantages to both approaches).
* matches all characters except a /, so *.jpg only expands to patterns in the current directory. **/ is similar, but also matches /, so it expands to patterns in any directory. This is supported by both bash and zsh.
So ls **/*.jpg will do what you want; you don't need to use find or grep. In zsh, especially you rarely need to use find since globbing is so much more powerful than in the standard Bourne shell or bash.
In zsh you can also use setopt glob_star_short and then **.jpg will work as well, which is a shortcut for **/*.jpg.
Bash is not recognizing the regular expression in this mv command:
mv ../downloads'^[exam].*$[.pdf] ../physics2400/exams
I'm trying to move files from a download directory to what ever directory I have made for them to go into.
An example of such a file is 'Exam 2 Practice Homework (Solutions).pdf'
(the single quotes are part of the file in Bash apparently.
There are many other files in the download folder hence the regex or the attempt anyway.
When performing filename expansion, Bash does not use regular expressions. Instead, a type of pattern matching referred to as globbing is used. This is discussed in the Filename Expansion section of the Bash manual.
In regards to your example file name (Exam 2 Practice Homework (Solutions).pdf), here are a couple things to note:
the single quotes are not part of the file name, but are a convenience to avoid having to escape special characters in the filename (i.e. the spaces and the parentheses). Without the quotes, the filename would be specified Exam\ 2\ Practice\ Homework\ \(Solutions\).pdf. See the Quoting section of the Bash manual for further details.
filesystems in Unix-like operating systems are case sensitive, so you need to account for the upper case E the filename starts with
Here's a pattern matching expression that would match your example filename as well as other files that start with Exam and end with .pdf.
mv ../downloads/Exam*.pdf ../phyiscs2400/exams
If you have files that start with both Exam and exam, you could account for both with the following:
mv ../downloads/[Ee]xam*.pdf ../phyiscs2400/exams
The bracketed expression is interpreted as "matches any one of the enclosed characters". This allows you to account for both upper and lower case.
Before executing such mv commands, I would test the filename expansion by running ls to verify that the intended files are matched:
ls ../downloads/[Ee]xam*.pdf
If you want to use the regular expression, how about this?
find ./downloads -regex '.*\.pdf' -exec mv '{}' exams/ \;
I was wondering how I can list files with ls in bash that will only list a specific subset of files?
For example, I have a folder with 10000 files, some of which are named:
temp_cc1_covmat and temp_cc1_slurm, but the values of 1 range from 1-1000.
So how would I list only say, temp_cc400_slurm-temp_cc_499_slurm?
I want to do this as I would like to queue files on a supercomputer that only ends with slurm. I could do sbatch *_slurm but there are also a lot of other files in the folder that ends with _slurm.
You can use this Brace Expansion in bash:
temp_cc{400..499}_slurm
To list these file use:
echo temp_cc{400..499}_slurm
or:
printf "%s\n" temp_cc{400..499}_slurm
or even ls:
ls temp_cc{400..499}_slurm
Using the ? wildcard:
$ ls temp_cc4??_slurm
man 7 glob:
Wildcard matching
A string is a wildcard pattern if it contains one of the characters
'?', '*' or '['. Globbing is the operation that expands a wildcard
pattern into the list of pathnames matching the pattern. Matching is
defined by:
A '?' (not between brackets) matches any single character.
The argument list too long error applies using the ? also. I tested with ls test????? and it worked but with ls test[12]????? I got the error. (Yes, you could ls temp_cc4[0-9][0-9]_slurm also.)
A command that prints a list of files and folders in the current directory along with their total sizes is du -sh *. That command alone doesn't, however, list hidden files or folders. I found a solution for a command that does correctly list the hidden files and folders along with the rest: du -sh .[!.]* *. Although it works perfectly, the solution was provided as-is, without any explanation.
What is the meaning of .[!.]*, exactly? How does it work?
It's a globbing pattern that basically tells bash to find all files starting with a ., followed by any character but a .and containing any character after that.
See this page for a great explanation of bash globbing patterns.
. - match a ., prefix of hidden file
[!.] - match any character, as long as it is not a ., see ref
* - any number of characters
so this pattern means match files starts with . but not ..
.[!.]* the meaning is any file or directory name start with . but not following with ., so it will include all hidden files and directories under current directory but exclude parent directory.
Because this behaviour is decided by shell glob pattern. So you can use ls .[!.]* to see what actually get in your shell environment.
BTW, you can turn dotglob on in your shell to simplify your du command.
$ shopt -s dotglob
$ du -sh *
$ shopt -u dotglob
From bash manual
dotglob If set, bash includes filenames beginning with a `.' in the results of pathname expansion.
I am trying the following command:
ls myfile.h1.{`seq -s ',' 3501 3511`}*
But ls raises the error:
ls: cannot access myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511}*: No such file or directory
Seems like ls is thinking the entire line is a filename and not a wildcard pattern. But if I just copy that command ls myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511}* in the terminal I get the listing as expected.
Why does typing out the command in full work, but not the usage with seq?
seq is not needed for your case, try
$ ls myfile.h1.{3500..3511}
if you want to use seq I would suggest using format option
$ ls $(seq -f 'myfile.h1.%g' 3501 3511)
but I don't think there is any reason to do so.
UPDATE:
Note that I didn't notice the globbing in the original post. With that, the brace extension still preferred way
$ ls myfile.h1.{3500..3511}*
perhaps even factoring the common digit out, if your bash support zero padding
$ ls myfile.h1.35{00..11}*
if not you can extract at least 3 out
$ ls myfile.h1.3{500..511}*
Note that the seq alternative won't work with globbing.
Other answer has more details...
karakfa's answer, which uses a literal sequence brace expansion expression, is the right solution.
As for why your approach didn't work:
Bash's brace expansion {...} only works with literal expressions - neither variable references nor, as in your case, command substitutions (`...`, or, preferably, $(...)) work[1] - for a concise overview, see this answer of mine.
With careful use of eval, however, you can work around this limitation; to wit:
from=3501 to=3511
# CAVEAT: Only do this if you TRUST that $from and $to contain
# decimal numbers only.
eval ls "myfile.h1.{$from..$to}*"
#ghoti suggests the following improvement in a comment to make the use of eval safe here:
# Use parameter expansion to remove all non-digit characters from the values
# of $from and $to, thus ensuring that they either contain only a decimal
# number or the empty string; this expansion happens *before* eval is invoked.
eval ls "myfile.h1.{${from//[^0-9]/}..${to//[^0-9]/}}*"
As for how your command was actually evaluated:
Note: Bash applies 7-8 kinds of expansions to a command line; only the ones that actually come into play here are discussed below.
first, the command in command substitution `seq -s ',' 3501 3511` is executed, and replaced by its output (also note the trailing ,):
3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,
the result then forms a single word with its prefix, myfile.h1.{ and its suffix, }*, yielding:
myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,}*
pathname expansion (globbing) is then applied to the result - in your case, since no files match, it is left as-is (by default; shell options shopt -s nullglob or shopt -s failglob could change that).
finally, literal myfile.h1.{3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,}* is passed to ls, which - because it doesn't refer to an existing filesystem item - results in the error message you saw.
[1] Note that the limitation only applies to sequence brace expansions (e.g., {1..3}); list brace expansions (e.g, {1,2,3}) are not affected, because no up-front interpretation (interpolation) is needed; e.g. {$HOME,$USER} works, because brace expansion results expanding the list to separate words $HOME, and $USER, which are only later expanded.
Historically, sequence brace expansions were introduced later, at a time when the order of shell expansions was already fixed.