I basically want to do the following bash command but in tcsh:
rm !(file1)
Thanks
You can use ls -1 (that's the number one, not the lowercase letter L) to list one file per line, and then use grep -vx <pattern> to exclude (-v) lines that exactly (-x) match <pattern>, and then xargs it to your command, rm. For example,
ls -1 | grep -vx file1 | xargs rm
In case your version of grep doesn't support the -x option, you can use anchors:
ls -1 | grep -vx '^file1$' | xargs rm
To use this with commands other than rm that may not take an arbitrary number of arguments, remember to add the -n 1 option to xargs so that arguments are handled one by one:
ls -1 | grep -vx '^file1$' | xargs -n 1 rm
I believe you can also achieve this using find's -name option to specify a parameter by negation, i.e. the find utility itself may support expressions like !(file1), though you'll still have to pipe the results to xargs.
tcsh has a special ^ syntax for glob patterns (not supported in csh, sh, or bash). Prefixing a glob pattern with ^ negates it, causing to match all file names that don't match the pattern.
Quoting the tcsh manual:
An entire glob-pattern can also be negated with `^':
> echo *
bang crash crunch ouch
> echo ^cr*
bang ouch
A single file name is not a glob pattern, and so the ^ prefix doesn't apply to it, but it can be turned into one by, for example, surrounding the first character with square brackets.
So this:
rm ^[f]ile1
should remove all files in the current directory other than file1.
I strongly recommend testing this before using it, either by using an echo command first:
echo ^[f]ile1
or by using Ctrl-X * to expand the pattern to a list of files before hitting Enter.
UPDATE: I've since learned that bash supports similar functionality but with a different syntax. In bash, !(PATTERN) matches anything not matched by the pattern. This is not recognized unless the extglob shell option is enabled. Unlike tcsh's ^ syntax, the pattern can be a single file name. This isn't relevant to what you're asking, but it could be useful if you ever decide to switch to bash.
zsh probably has something similar.
Related
I have a folder named "components" and within that folder a file name "apple"
If I cd to "components" folder and execute the following command:
ls | grep -G a*e
It works and returns apple correctly.
However, if I do not cd to components folder and execute the following command:
ls components | grep -G a*e
It does not work and returns blank. What could be the reason?
A third grep command below works fine.
ls components | grep ap
The actual filename I am grepping is complex. So I need the grep -G tag to work.
Unquoted, a*e is a shell glob pattern that is expanded by the shell before grep runs.
When you are in the directory, this:
ls | grep -G a*e
becomes
ls | grep -G apple
As you have a file named 'apple' this matches.
When you are not in the folder, and you run:
ls components | grep -G a*e
the shell again attempts to expand the glob pattern.
If there is any file in your current directory that matches (for example, "abalone"), then the glob will expand to that. It may expand to multiple strings if there is more than one such filename (for example, "abalone", "algae"). The command becomes something like:
ls components | grep -G abalone
ls components | grep -G abalone algae
In the first case, you will get blank output unless components directory also contains that filename.
In the second case, grep will ignore the directory entirely and attempt to find the string "abalone" inside the file "algae".
There is a third possibility: the glob fails to find anything. In this case, grep will receive the regexp a*e. The -G option to grep tell it to use BRE-style regexp. With these, a*e means "zero or more a followed by e". This is equivalent to saying "contains e".
In that case, you should see apple in your results regardless of whether you are in components or not. In a comment, you say that ls components | grep "a*e" returned nothing. As quoting should force precisely the same result as this third case, this is surprising.
Note that if you are intending to use globs you don't need grep at all:
cd components
ls a*e
ls components/a*e
a*e is a glob, not a regex. It's important to understand the difference.
The shell expands globs in unquoted arguments by matching the argument with available files. The * in a*e means "any sequence of characters not containing a directory separator", so it will match the filename apple (or accolade.node) as long as that file is present in the current directory. Glob matches are complete, not substring matches.
So when you execute grep a*e in a directory which contains the file apple, the shell will replace a*e with the word apple before invoking grep, making the command grep apple. If the directory also contained the file accolade.node, the shell would have put that into the command line as well; grep accolade.node apple. That's very rarely what you want to happen to grep arguments (other than filename arguments), so it's highly recommended to get into the habit of quoting arguments.
Unlike the shell, grep is based on regular expression matching. In a regular expression, * means "any number of repetitions of the previous element", so the regular expression a*e will match e, ae, aae, aaae, and so on. Since grep does substring matching (by default), those strings could be anywhere in the line being matched. That will match the e in apple, for example, but it will also match any other line which contains an e, such as electronics. (That makes it a bit surprising that ls components | grep "a*e" did not match components/apple. Perhaps there was some typing problem.)
In order to match a followed by a sequence of arbitrary characters followed by an e, you could use the regular expression a.*e (i.e. grep "a.*e" -- note the use of quotes to avoid having the shell try to expand that argument as a glob). But that will probably match too much, if you're expecting it to do the same thing as the glob a*e. You might want to add some restrictions. For example, grep -w forces the match to be complete words. And (with gnu grep, at least) you can use grep -w "a\S*e" to match a complete word which starts with a and ends with e, using the \S shortcut (any character other than whitespace).
You very rarely want to use -G, by the way, particularly since it's the default (unfortunately). Most of the time, you'll want to use grep -E in order to not have to insert backslashes throughout your pattern. Please read man 7 regex for a quick overview of regex syntax and the difference between basic and extended Posix regexes. man grep is also useful, of course.
I want to search a certain string in a number of archival log folders which reflect different servers. I use 2 different commands as of now
-bash-4.1$ zcat /mnt/bkp/logs/cmmt-54-22[8-9]/my_app.2021-12-28-* | grep 'abc'
and
-bash-4.1$ zcat /mnt/bkp/logs/cmmt-54-23[0-3]/my_app.2021-12-28-* | grep 'abc'
I basically want to search on server folders cmmt-54-228, cmmt-54-229 .... cmmt-54-233.
I tried combining the two commands into one but it doesn't seem to be working some mistake in using regex from my side
-bash-4.1$ zcat /mnt/bkp/logs/cmmt-54-22[8-9]|3[0-3]/my_app.2021-12-28-* | grep 'abc'
Please help.
Regex is not glob. See man 7 glob vs man 7 regex.
grep with with regex. grep filters lines that match some regular expresion.
Shell expands words that you write. Shell replaces what you write that contains "filename expansion triggers" * ? [ and replaces that word with a list of words of matching filenames.
You can use extended pattern matching (see man bash), which sounds like the most natural here:
shopt -s extglob
echo /mnt/bkp/logs/cmmt-54-2#(2[8-9]|3[0-3])/my_app.2021-12-28-*
In interactive shell I would just write it twice:
zcat /mnt/bkp/logs/cmmt-54-22[8-9]/my_app.2021-12-28-* /mnt/bkp/logs/cmmt-54-23[0-3]/my_app.2021-12-28-*
Or with brace expansion (see man bash):
zcat /mnt/bkp/logs/cmmt-54-2{2[8-9],3[0-3]}/my_app.2021-12-28-*
Braces expansion first replaces the word by two words, then filename expansion replaces them for actual filenames.
You can also find files with a -regex. For that, see man find. (Or output a list of filenames and pipe it to grep and then use xargs or similar to pass it to a command)
I am new with shell script.
I need to save the number of files with particular extension(.properties) in a variable using shell script.
I have used
ls |grep .properties$ |wc -l
but this command prints the number of properties files in the folder. How can I assign this value in a variable.
I have tried
count=${ls |grep .properties$ |wc -l}
But it is showing error like:
./replicate.sh: line 57: ${ls |grep .properties$ |wc -l}: bad substitution
What is this type of errors?
Please anyone help me to save the number of particular files in a variable for future use.
You're using the wrong brackets, it should be $() (command output substitution) rather than ${} (variable substitution).
count=$(ls -1 | grep '\.properties$' | wc -l)
You'll also notice I've use ls -1 to force one file per line in case your ls doesn't do this automatically for pipelines, and changed the pattern to match the . correctly.
You can also bypass the grep totally if you use something like:
count=$(ls -1 *.properties 2>/dev/null | wc -l)
Just watch out for "evil" filenames like those with embedded newlines for example, though my ls seems to handle these fine by replacing the newline with a ? character - that's not necessarily a good idea for doing things with files but it works okay for counting them.
There are better tools to use if you have such beasts and you need the actual file name, but they're rare enough that you generally don't have to worry about it.
You could use a loop with globbing:
count=0
for i in *.properties; do
count=$((count+1))
done
If you are using a shell that supports arrays, you can simply capture all such file names
files=( *.properties )
and then determine the number of array elements
count=${#files[#]}
(The above assumes bash; other shells may require slightly different syntax.)
You'd better use find instead of parsing ls. Then, use the var=$(command) syntax to store the value.
var=$(find . -maxdepth 1 -name "*\.properties" | wc -l)
Reference: Why you shouldn't parse the output of ls.
To solve the problem appearing if any file name contains new lines, you can use what chepner suggests in the comments:
var=$(find . -maxdepth 1 -name "*\.properties" -exec 'echo 1' | wc -l)
so that for every match it will print not the name, but any random character (in this case, 1) and then the amount of them will be counted to produce the correct output.
Use:
count=`ls|grep .properties$ | wc -l`
echo $count
You could write your assignment like this:
count=$(ls -q | grep -c '\.properties$')
or
count=$(ls -qA | grep -c '\.properties$')
if you want to include hidden files.
This works with all kind of filenames because we're using ls with q.
Sure it's easier to link to some webpage that tells you to "never parse ls" than to read the ls manual and see there's a q option (and that most implementations default to q if the output is to a terminal device which explains why some people here state their ls seems to handle filenames with newlines just fine by replacing the newline with a ? character).
Objective
Change these filenames:
F00001-0708-RG-biasliuyda
F00001-0708-CS-akgdlaul
F00001-0708-VF-hioulgigl
to these filenames:
F0001-0708-RG-biasliuyda
F0001-0708-CS-akgdlaul
F0001-0708-VF-hioulgigl
Shell Code
To test:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/'
To perform:
ls F00001-0708-*|sed 's/\(.\).\(.*\)/mv & \1\2/' | sh
My Question
I don't understand the sed code. I understand what the substitution
command
$ sed 's/something/mv'
means. And I understand regular expressions somewhat. But I don't
understand what's happening here:
\(.\).\(.*\)
or here:
& \1\2/
The former, to me, just looks like it means: "a single character,
followed by a single character, followed by any length sequence of a
single character"--but surely there's more to it than that. As far as
the latter part:
& \1\2/
I have no idea.
First, I should say that the easiest way to do this is to use the
prename or rename commands.
On Ubuntu, OSX (Homebrew package rename, MacPorts package p5-file-rename), or other systems with perl rename (prename):
rename s/0000/000/ F0000*
or on systems with rename from util-linux-ng, such as RHEL:
rename 0000 000 F0000*
That's a lot more understandable than the equivalent sed command.
But as for understanding the sed command, the sed manpage is helpful. If
you run man sed and search for & (using the / command to search),
you'll find it's a special character in s/foo/bar/ replacements.
s/regexp/replacement/
Attempt to match regexp against the pattern space. If success‐
ful, replace that portion matched with replacement. The
replacement may contain the special character & to refer to that
portion of the pattern space which matched, and the special
escapes \1 through \9 to refer to the corresponding matching
sub-expressions in the regexp.
Therefore, \(.\) matches the first character, which can be referenced by \1.
Then . matches the next character, which is always 0.
Then \(.*\) matches the rest of the filename, which can be referenced by \2.
The replacement string puts it all together using & (the original
filename) and \1\2 which is every part of the filename except the 2nd
character, which was a 0.
This is a pretty cryptic way to do this, IMHO. If for
some reason the rename command was not available and you wanted to use
sed to do the rename (or perhaps you were doing something too complex
for rename?), being more explicit in your regex would make it much
more readable. Perhaps something like:
ls F00001-0708-*|sed 's/F0000\(.*\)/mv & F000\1/' | sh
Being able to see what's actually changing in the
s/search/replacement/ makes it much more readable. Also it won't keep
sucking characters out of your filename if you accidentally run it
twice or something.
you've had your sed explanation, now you can use just the shell, no need external commands
for file in F0000*
do
echo mv "$file" "${file/#F0000/F000}"
# ${file/#F0000/F000} means replace the pattern that starts at beginning of string
done
I wrote a small post with examples on batch renaming using sed couple of years ago:
http://www.guyrutenberg.com/2009/01/12/batch-renaming-using-sed/
For example:
for i in *; do
mv "$i" "`echo $i | sed "s/regex/replace_text/"`";
done
If the regex contains groups (e.g. \(subregex\) then you can use them in the replacement text as \1\,\2 etc.
The easiest way would be:
for i in F00001*; do mv "$i" "${i/F00001/F0001}"; done
or, portably,
for i in F00001*; do mv "$i" "F0001${i#F00001}"; done
This replaces the F00001 prefix in the filenames with F0001.
credits to mahesh here: http://www.debian-administration.org/articles/150
The sed command
s/\(.\).\(.*\)/mv & \1\2/
means to replace:
\(.\).\(.*\)
with:
mv & \1\2
just like a regular sed command. However, the parentheses, & and \n markers change it a little.
The search string matches (and remembers as pattern 1) the single character at the start, followed by a single character, follwed by the rest of the string (remembered as pattern 2).
In the replacement string, you can refer to these matched patterns to use them as part of the replacement. You can also refer to the whole matched portion as &.
So what that sed command is doing is creating a mv command based on the original file (for the source) and character 1 and 3 onwards, effectively removing character 2 (for the destination). It will give you a series of lines along the following format:
mv F00001-0708-RG-biasliuyda F0001-0708-RG-biasliuyda
mv abcdef acdef
and so on.
Using perl rename (a must have in the toolbox):
rename -n 's/0000/000/' F0000*
Remove -n switch when the output looks good to rename for real.
There are other tools with the same name which may or may not be able to do this, so be careful.
The rename command that is part of the util-linux package, won't.
If you run the following command (GNU)
$ rename
and you see perlexpr, then this seems to be the right tool.
If not, to make it the default (usually already the case) on Debian and derivative like Ubuntu :
$ sudo apt install rename
$ sudo update-alternatives --set rename /usr/bin/file-rename
For archlinux:
pacman -S perl-rename
For RedHat-family distros:
yum install prename
The 'prename' package is in the EPEL repository.
For Gentoo:
emerge dev-perl/rename
For *BSD:
pkg install gprename
or p5-File-Rename
For Mac users:
brew install rename
If you don't have this command with another distro, search your package manager to install it or do it manually:
cpan -i File::Rename
Old standalone version can be found here
man rename
This tool was originally written by Larry Wall, the Perl's dad.
The backslash-paren stuff means, "while matching the pattern, hold on to the stuff that matches in here." Later, on the replacement text side, you can get those remembered fragments back with "\1" (first parenthesized block), "\2" (second block), and so on.
If all you're really doing is removing the second character, regardless of what it is, you can do this:
s/.//2
but your command is building a mv command and piping it to the shell for execution.
This is no more readable than your version:
find -type f | sed -n 'h;s/.//4;x;s/^/mv /;G;s/\n/ /g;p' | sh
The fourth character is removed because find is prepending each filename with "./".
Here's what I would do:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done
Then if that looks ok, add | sh to the end. So:
for file in *.[Jj][Pp][Gg] ;do
echo mv -vi \"$file\" `jhead $file|
grep Date|
cut -b 16-|
sed -e 's/:/-/g' -e 's/ /_/g' -e 's/$/.jpg/g'` ;
done | sh
for i in *; do mv $i $(echo $i|sed 's/AAA/BBB/'); done
The parentheses capture particular strings for use by the backslashed numbers.
ls F00001-0708-*|sed 's|^F0000\(.*\)|mv & F000\1|' | bash
Some examples that work for me:
$ tree -L 1 -F .
.
├── A.Show.2020.1400MB.txt
└── Some Show S01E01 the Loreming.txt
0 directories, 2 files
## remove "1400MB" (I: ignore case) ...
$ for f in *; do mv 2>/dev/null -v "$f" "`echo $f | sed -r 's/.[0-9]{1,}mb//I'`"; done;
renamed 'A.Show.2020.1400MB.txt' -> 'A.Show.2020.txt'
## change "S01E01 the" to "S01E01 The"
## \U& : change (here: regex-selected) text to uppercase;
## note also: no need here for `\1` in that regex expression
$ for f in *; do mv 2>/dev/null "$f" "`echo $f | sed -r "s/([0-9] [a-z])/\U&/"`"; done
$ tree -L 1 -F .
.
├── A.Show.2020.txt
└── Some Show S01E01 The Loreming.txt
0 directories, 2 files
$
2>/dev/null suppresses extraneous output (warnings ...)
reference [this thread]: https://stackoverflow.com/a/2372808/1904943
change case: https://www.networkworld.com/article/3529409/converting-between-uppercase-and-lowercase-on-the-linux-command-line.html
I know you can do it with a find, but is there a way to send the output of ls to mv in the unix command line?
ls is a tool used to DISPLAY some statistics about filenames in a directory.
It is not a tool you should use to enumerate them and pass them to another tool for using it there. Parsing ls is almost always the wrong thing to do, and it is bugged in many ways.
For a detailed document on the badness of parsing ls, which you should really go read, check out: http://mywiki.wooledge.org/ParsingLs
Instead, you should use either globs or find, depending on what exactly you're trying to achieve:
mv * /foo
find . -exec mv {} /foo \;
The main source of badness of parsing ls is that ls dumps all filenames into a single string of output, and there is no way to tell the filenames apart from there. For all you know, the entire ls output could be one single filename!
The secondary source of badness of parsing ls comes from the broken way in which half the world uses bash. They think for magically does what they would like it to do when they do something like:
for file in `ls` # Never do this!
for file in $(ls) # Exactly the same thing.
for is a bash builtin that iterates over arguments. And $(ls) takes the output of ls and cuts it apart into arguments wherever there are spaces, newlines or tabs. Which basically means, you're iterating over words, not over filenames. Even worse, you're asking bask to take each of those mutilated filename words and then treat them as globs that may match filenames in the current directory. So if you have a filename which contains a word which happens to be a glob that matches other filenames in the current directory, that word will disappear and all those matching filenames will appear in its stead!
mv `ls` /foo # Exact same badness as the ''for'' thing.
One way is with backticks:
mv `ls *.boo` subdir
Edit: however, this is fragile and not recommended -- see #lhunath's asnwer for detailed explanations and recommendations.
None of the answers so far are safe for filenames with spaces in them. Try this:
for i in *; do mv "$i" some_dir/; done
You can of course use any glob pattern you like in place of *.
Not exactly sure what you're trying to achieve here, but here's one possibility:
The "xargs" part is the important piece everything else is just setup. The effect of this is to take everything that "ls" outputs and add a ".txt" extension to it.
$ mkdir xxx #
$ cd xxx
$ touch a b c x y z
$ ls
a b c x y z
$ ls | xargs -Ifile mv file file.txt
$ ls
a.txt b.txt c.txt x.txt y.txt z.txt
$
Something like this could also be achieved by:
$ touch a b c x y z
$ for i in `ls`;do mv $i ${i}.txt; done
$ ls
a.txt b.txt c.txt x.txt y.txt z.txt
$
I sort of like the second way better. I can NEVER remember how xargs works without reading the man page or going to my "cute tricks" file.
Hope this helps.
Check out find -exec {} as it might be a better option than ls but it depends on what you're trying to achieve.
/bin/ls | tr '\n' '\0' | xargs -0 -i% mv % /path/to/destdir/
"Useless use of ls", but should work. By specifying the full path to ls(1) you avoid clashes with aliasing of ls(1) mentioned in some of the previous posts. The tr(1) command together with "xargs -0" makes the command work with filenames containing (ugh) whitespace. It won't work with filenames containing newlines, but having filenames like that in the file system is to ask for trouble, so it probably won't be a big problem. But filenames with newlines could exist, so a better solution would be to use "find -print0":
find /path/to/srcdir -type f -print0 | xargs -0 -i% mv % dest/
You shouldn't use the output of ls as the input of another command. Files with spaces in their names are difficult as is the inclusion of ANSI escape sequences if you have:
alias ls-'ls --color=always'
for example.
Always use find or xargs (with -0) or globbing.
Also, you didn't say whether you want to move files or rename them. Each would be handled differently.
edit: added -0 to xargs (thanks for the reminder)
Backticks work well, as others have suggested. See xargs, too. And for really complicated stuff, pipe it into sed, make the list of commands you want, then run it again with the output of sed piped into sh.
Here's an example with find, but it works fine with ls, too:
http://github.com/DonBranson/scripts/blob/f09d24629ab6eb3ce509d4d3078818430306b063/jarfinder.sh
#!/bin/bash
for i in $( ls * );
do
mv $1 /backup/$1
done
else, it's the find solution by sybreon, and as suggested NOT the green mv ls solution.
Just use find or your shells globing!
find . -depth=1 -exec mv {} /tmp/blah/ \;
..or..
mv * /tmp/blah/
You don't have to worry about colour in the ls output, or other piping strangeness - Linux allows basically any characters in the filename except a null byte.. For example:
$ touch "blah\new|
> "
$ ls | xargs file
blahnew|: cannot open `blahnew|' (No such file or directory)
..but find works perfectly:
$ find . -exec file {} \;
./blah\new|
: empty
So this answer doesn't send the output of ls to mv but as #lhunath explained using ls is almost always the wrong tool for the job. Use shell globs or a find command.
For more complicated cases (often in a script), using bash arrays to build up the argument list from shell globs or find commands can be very useful. One can create an array and push to it with the appropriate conditional logic. This also handles spaces in filenames properly.
For example:
myargs=()
# don't push if the glob does not match anything
shopt -s nullglob
myargs+=(myfiles*)
To push files matching a find to the array: https://stackoverflow.com/a/23357277/430128.
The last argument should be the target location:
myargs+=("Some target directory")
Use myargs in the invocation of a command like mv:
mv "${myargs[#]}"
Note the quoting of the array myargs to pass array elements with spaces correctly.
You surround the ls with back quotes and put it after the mv, so like this...
mv `ls` somewhere/
But keep in mind that if any of your file names have spaces in them it won't work very well.
Also it would be simpler to just do something like this: mv filepattern* somewhere/