If I do
$ ls -l --color=always
I get a list of files inside the directory with some nice colouring for different file types etc..
Now, I want to be able to pipe the coloured output of ls through grep to filter out some files I don't need. The key is that I still want to preserve the colouring after the grep filter.
$ ls -l --color=always | grep -E some_regex
^ I lose the colouring after grep
EDIT: I'm using headless-server Ubuntu 8.10, Bash 3.2.39, pretty much a stock install with no fancy configs
Your grep is probably removing ls's color codes because it has its own coloring turned on.
You "could" do this:
ls -l --color=always | grep --color=never pattern
However, it is very important that you understand what exactly you're grepping here. Not only is grepping ls unnecessary (use a glob instead), this particular case is grepping through not only filenames and file stats, but also through the color codes added by ls!
The real answer to your question is: Don't grep it. There is never a need to pipe ls into anything or capture its output. ls is only intended for human interpretation (eg. to look at in an interactive shell only, and for this purpose it is extremely handy, of course). As mentioned before, you can filter what files ls enumerates by using globs:
ls -l *.txt # Show all files with filenames ending with `.txt'.
ls -l !(foo).txt # Show all files with filenames that end on `.txt' but aren't `foo.txt'. (This requires `shopt -s extglob` to be on, you can put it in ~/.bashrc)
I highly recommend you read these two excellent documents on the matter:
Explanation of the badness of parsing ls: http://mywiki.wooledge.org/ParsingLs
The power of globs: http://mywiki.wooledge.org/glob
You should check if you are really using the "real" ls, just by directly calling the binary:
/bin/ls ....
Because: The code you described really should work, unless ls ignores --color=always for some weird reason or bug.
I suspect some alias or function that adds (directly or through a variable) some options. Double-check that this isn't the case.
Related
I have an assignment where, only using bash one-liners, I must ls the specific directories in my home directory that do not follow a specific naming schema. In my home directory, there are some directories that have the format of 3 alphabetical lower case letters followed by 3 decimal digits. However, there are other directories that don't follow this format. I must list those files and output the info to a txt file. Here are some commands I have written so far and am experimenting with:
ls /home -1 | sed [^a-z][^a-z][^a-z].[^0-9][^0-9][^0-9]
ls /home -1 "[^[a-z][a-z][a-z][0-9][0-9][0-9]]"
ls /home -1 *{[^a-z][^a-z][^a-z].[^0-9][^0-9][^0-9]}*
Also before anyone asks, I know formatting and searching through the output of the ls command is not as effective as the find command. But the assignment that I am working on dictates that I may only use these commands: ls, ps, sed, cut, paste, sort, tr, grep, awk, cat, uniq
If you can use shopt -s extglob first, then
ls -1d /home/!([a-z][a-z][a-z][0-9][0-9][0-9])
If not,
ls -1d /home/* | grep -v '/home/[a-z][a-z][a-z][0-9][0-9][0-9]$'
PLEASE read the manual pages for ls, grep, and shopt.
If you don't understand why these work, you haven't learned anything, and we're just doing your work for you...
Fellows, I have a .sh that creates a file.log, in this file I have many git logs searched from a range of dates that the user passed before (just to explain what its have).
Now I need to use this file.log that I have from a this external program and use with this code:
find ./* -type f -exec grep -l 'a1009206_vcr' {} \; > file.log
my question is how can I do this?
Recurse with Grep While Ignoring Missing/Unreadable Files
The BSD and GNU grep utilities have options that can save you the hassle of using find, xargs, et al. in many cases. This is one of those. For example:
grep -Flrs "a1009206_vcr" . > file.log
This uses the following flags:
-F, --fixed-strings
Interpret pattern as a set of fixed strings (i.e. force grep to
behave as fgrep).
-l, --files-with-matches
Only the names of files containing selected lines are written to
standard output. grep will only search a file until a match has
been found, making searches potentially less expensive. Path-
names are listed once per file searched. If the standard input
is searched, the string ``(standard input)'' is written.
-R, -r, --recursive
Recursively search subdirectories listed.
-s, --no-messages
Silent mode. Nonexistent and unreadable files are ignored (i.e.
their error messages are suppressed).
to recurse down through the present working directory (e.g. . or $PWD if you prefer) and writing a list of filenames with matches to file.log. The -s flag keeps permissions errors or other cruft from cluttering your output file. You can also turn off standard error with 2>&- if you're so inclined.
Caveat: Symlinks and Recursion
The above should work in most cases, but you may also need to add either -O or -p if you're recursing and don't want to follow some or all of your symlinks. The man page has more specifics about grep's default behavior regarding symlinks, with and without recursing.
If I do
$ ls -l --color=always
I get a list of files inside the directory with some nice colouring for different file types etc..
Now, I want to be able to pipe the coloured output of ls through grep to filter out some files I don't need. The key is that I still want to preserve the colouring after the grep filter.
$ ls -l --color=always | grep -E some_regex
^ I lose the colouring after grep
EDIT: I'm using headless-server Ubuntu 8.10, Bash 3.2.39, pretty much a stock install with no fancy configs
Your grep is probably removing ls's color codes because it has its own coloring turned on.
You "could" do this:
ls -l --color=always | grep --color=never pattern
However, it is very important that you understand what exactly you're grepping here. Not only is grepping ls unnecessary (use a glob instead), this particular case is grepping through not only filenames and file stats, but also through the color codes added by ls!
The real answer to your question is: Don't grep it. There is never a need to pipe ls into anything or capture its output. ls is only intended for human interpretation (eg. to look at in an interactive shell only, and for this purpose it is extremely handy, of course). As mentioned before, you can filter what files ls enumerates by using globs:
ls -l *.txt # Show all files with filenames ending with `.txt'.
ls -l !(foo).txt # Show all files with filenames that end on `.txt' but aren't `foo.txt'. (This requires `shopt -s extglob` to be on, you can put it in ~/.bashrc)
I highly recommend you read these two excellent documents on the matter:
Explanation of the badness of parsing ls: http://mywiki.wooledge.org/ParsingLs
The power of globs: http://mywiki.wooledge.org/glob
You should check if you are really using the "real" ls, just by directly calling the binary:
/bin/ls ....
Because: The code you described really should work, unless ls ignores --color=always for some weird reason or bug.
I suspect some alias or function that adds (directly or through a variable) some options. Double-check that this isn't the case.
I am new with shell script.
I need to save the number of files with particular extension(.properties) in a variable using shell script.
I have used
ls |grep .properties$ |wc -l
but this command prints the number of properties files in the folder. How can I assign this value in a variable.
I have tried
count=${ls |grep .properties$ |wc -l}
But it is showing error like:
./replicate.sh: line 57: ${ls |grep .properties$ |wc -l}: bad substitution
What is this type of errors?
Please anyone help me to save the number of particular files in a variable for future use.
You're using the wrong brackets, it should be $() (command output substitution) rather than ${} (variable substitution).
count=$(ls -1 | grep '\.properties$' | wc -l)
You'll also notice I've use ls -1 to force one file per line in case your ls doesn't do this automatically for pipelines, and changed the pattern to match the . correctly.
You can also bypass the grep totally if you use something like:
count=$(ls -1 *.properties 2>/dev/null | wc -l)
Just watch out for "evil" filenames like those with embedded newlines for example, though my ls seems to handle these fine by replacing the newline with a ? character - that's not necessarily a good idea for doing things with files but it works okay for counting them.
There are better tools to use if you have such beasts and you need the actual file name, but they're rare enough that you generally don't have to worry about it.
You could use a loop with globbing:
count=0
for i in *.properties; do
count=$((count+1))
done
If you are using a shell that supports arrays, you can simply capture all such file names
files=( *.properties )
and then determine the number of array elements
count=${#files[#]}
(The above assumes bash; other shells may require slightly different syntax.)
You'd better use find instead of parsing ls. Then, use the var=$(command) syntax to store the value.
var=$(find . -maxdepth 1 -name "*\.properties" | wc -l)
Reference: Why you shouldn't parse the output of ls.
To solve the problem appearing if any file name contains new lines, you can use what chepner suggests in the comments:
var=$(find . -maxdepth 1 -name "*\.properties" -exec 'echo 1' | wc -l)
so that for every match it will print not the name, but any random character (in this case, 1) and then the amount of them will be counted to produce the correct output.
Use:
count=`ls|grep .properties$ | wc -l`
echo $count
You could write your assignment like this:
count=$(ls -q | grep -c '\.properties$')
or
count=$(ls -qA | grep -c '\.properties$')
if you want to include hidden files.
This works with all kind of filenames because we're using ls with q.
Sure it's easier to link to some webpage that tells you to "never parse ls" than to read the ls manual and see there's a q option (and that most implementations default to q if the output is to a terminal device which explains why some people here state their ls seems to handle filenames with newlines just fine by replacing the newline with a ? character).
I know you can do it with a find, but is there a way to send the output of ls to mv in the unix command line?
ls is a tool used to DISPLAY some statistics about filenames in a directory.
It is not a tool you should use to enumerate them and pass them to another tool for using it there. Parsing ls is almost always the wrong thing to do, and it is bugged in many ways.
For a detailed document on the badness of parsing ls, which you should really go read, check out: http://mywiki.wooledge.org/ParsingLs
Instead, you should use either globs or find, depending on what exactly you're trying to achieve:
mv * /foo
find . -exec mv {} /foo \;
The main source of badness of parsing ls is that ls dumps all filenames into a single string of output, and there is no way to tell the filenames apart from there. For all you know, the entire ls output could be one single filename!
The secondary source of badness of parsing ls comes from the broken way in which half the world uses bash. They think for magically does what they would like it to do when they do something like:
for file in `ls` # Never do this!
for file in $(ls) # Exactly the same thing.
for is a bash builtin that iterates over arguments. And $(ls) takes the output of ls and cuts it apart into arguments wherever there are spaces, newlines or tabs. Which basically means, you're iterating over words, not over filenames. Even worse, you're asking bask to take each of those mutilated filename words and then treat them as globs that may match filenames in the current directory. So if you have a filename which contains a word which happens to be a glob that matches other filenames in the current directory, that word will disappear and all those matching filenames will appear in its stead!
mv `ls` /foo # Exact same badness as the ''for'' thing.
One way is with backticks:
mv `ls *.boo` subdir
Edit: however, this is fragile and not recommended -- see #lhunath's asnwer for detailed explanations and recommendations.
None of the answers so far are safe for filenames with spaces in them. Try this:
for i in *; do mv "$i" some_dir/; done
You can of course use any glob pattern you like in place of *.
Not exactly sure what you're trying to achieve here, but here's one possibility:
The "xargs" part is the important piece everything else is just setup. The effect of this is to take everything that "ls" outputs and add a ".txt" extension to it.
$ mkdir xxx #
$ cd xxx
$ touch a b c x y z
$ ls
a b c x y z
$ ls | xargs -Ifile mv file file.txt
$ ls
a.txt b.txt c.txt x.txt y.txt z.txt
$
Something like this could also be achieved by:
$ touch a b c x y z
$ for i in `ls`;do mv $i ${i}.txt; done
$ ls
a.txt b.txt c.txt x.txt y.txt z.txt
$
I sort of like the second way better. I can NEVER remember how xargs works without reading the man page or going to my "cute tricks" file.
Hope this helps.
Check out find -exec {} as it might be a better option than ls but it depends on what you're trying to achieve.
/bin/ls | tr '\n' '\0' | xargs -0 -i% mv % /path/to/destdir/
"Useless use of ls", but should work. By specifying the full path to ls(1) you avoid clashes with aliasing of ls(1) mentioned in some of the previous posts. The tr(1) command together with "xargs -0" makes the command work with filenames containing (ugh) whitespace. It won't work with filenames containing newlines, but having filenames like that in the file system is to ask for trouble, so it probably won't be a big problem. But filenames with newlines could exist, so a better solution would be to use "find -print0":
find /path/to/srcdir -type f -print0 | xargs -0 -i% mv % dest/
You shouldn't use the output of ls as the input of another command. Files with spaces in their names are difficult as is the inclusion of ANSI escape sequences if you have:
alias ls-'ls --color=always'
for example.
Always use find or xargs (with -0) or globbing.
Also, you didn't say whether you want to move files or rename them. Each would be handled differently.
edit: added -0 to xargs (thanks for the reminder)
Backticks work well, as others have suggested. See xargs, too. And for really complicated stuff, pipe it into sed, make the list of commands you want, then run it again with the output of sed piped into sh.
Here's an example with find, but it works fine with ls, too:
http://github.com/DonBranson/scripts/blob/f09d24629ab6eb3ce509d4d3078818430306b063/jarfinder.sh
#!/bin/bash
for i in $( ls * );
do
mv $1 /backup/$1
done
else, it's the find solution by sybreon, and as suggested NOT the green mv ls solution.
Just use find or your shells globing!
find . -depth=1 -exec mv {} /tmp/blah/ \;
..or..
mv * /tmp/blah/
You don't have to worry about colour in the ls output, or other piping strangeness - Linux allows basically any characters in the filename except a null byte.. For example:
$ touch "blah\new|
> "
$ ls | xargs file
blahnew|: cannot open `blahnew|' (No such file or directory)
..but find works perfectly:
$ find . -exec file {} \;
./blah\new|
: empty
So this answer doesn't send the output of ls to mv but as #lhunath explained using ls is almost always the wrong tool for the job. Use shell globs or a find command.
For more complicated cases (often in a script), using bash arrays to build up the argument list from shell globs or find commands can be very useful. One can create an array and push to it with the appropriate conditional logic. This also handles spaces in filenames properly.
For example:
myargs=()
# don't push if the glob does not match anything
shopt -s nullglob
myargs+=(myfiles*)
To push files matching a find to the array: https://stackoverflow.com/a/23357277/430128.
The last argument should be the target location:
myargs+=("Some target directory")
Use myargs in the invocation of a command like mv:
mv "${myargs[#]}"
Note the quoting of the array myargs to pass array elements with spaces correctly.
You surround the ls with back quotes and put it after the mv, so like this...
mv `ls` somewhere/
But keep in mind that if any of your file names have spaces in them it won't work very well.
Also it would be simpler to just do something like this: mv filepattern* somewhere/