Fellows, I have a .sh that creates a file.log, in this file I have many git logs searched from a range of dates that the user passed before (just to explain what its have).
Now I need to use this file.log that I have from a this external program and use with this code:
find ./* -type f -exec grep -l 'a1009206_vcr' {} \; > file.log
my question is how can I do this?
Recurse with Grep While Ignoring Missing/Unreadable Files
The BSD and GNU grep utilities have options that can save you the hassle of using find, xargs, et al. in many cases. This is one of those. For example:
grep -Flrs "a1009206_vcr" . > file.log
This uses the following flags:
-F, --fixed-strings
Interpret pattern as a set of fixed strings (i.e. force grep to
behave as fgrep).
-l, --files-with-matches
Only the names of files containing selected lines are written to
standard output. grep will only search a file until a match has
been found, making searches potentially less expensive. Path-
names are listed once per file searched. If the standard input
is searched, the string ``(standard input)'' is written.
-R, -r, --recursive
Recursively search subdirectories listed.
-s, --no-messages
Silent mode. Nonexistent and unreadable files are ignored (i.e.
their error messages are suppressed).
to recurse down through the present working directory (e.g. . or $PWD if you prefer) and writing a list of filenames with matches to file.log. The -s flag keeps permissions errors or other cruft from cluttering your output file. You can also turn off standard error with 2>&- if you're so inclined.
Caveat: Symlinks and Recursion
The above should work in most cases, but you may also need to add either -O or -p if you're recursing and don't want to follow some or all of your symlinks. The man page has more specifics about grep's default behavior regarding symlinks, with and without recursing.
Related
If I do
$ ls -l --color=always
I get a list of files inside the directory with some nice colouring for different file types etc..
Now, I want to be able to pipe the coloured output of ls through grep to filter out some files I don't need. The key is that I still want to preserve the colouring after the grep filter.
$ ls -l --color=always | grep -E some_regex
^ I lose the colouring after grep
EDIT: I'm using headless-server Ubuntu 8.10, Bash 3.2.39, pretty much a stock install with no fancy configs
Your grep is probably removing ls's color codes because it has its own coloring turned on.
You "could" do this:
ls -l --color=always | grep --color=never pattern
However, it is very important that you understand what exactly you're grepping here. Not only is grepping ls unnecessary (use a glob instead), this particular case is grepping through not only filenames and file stats, but also through the color codes added by ls!
The real answer to your question is: Don't grep it. There is never a need to pipe ls into anything or capture its output. ls is only intended for human interpretation (eg. to look at in an interactive shell only, and for this purpose it is extremely handy, of course). As mentioned before, you can filter what files ls enumerates by using globs:
ls -l *.txt # Show all files with filenames ending with `.txt'.
ls -l !(foo).txt # Show all files with filenames that end on `.txt' but aren't `foo.txt'. (This requires `shopt -s extglob` to be on, you can put it in ~/.bashrc)
I highly recommend you read these two excellent documents on the matter:
Explanation of the badness of parsing ls: http://mywiki.wooledge.org/ParsingLs
The power of globs: http://mywiki.wooledge.org/glob
You should check if you are really using the "real" ls, just by directly calling the binary:
/bin/ls ....
Because: The code you described really should work, unless ls ignores --color=always for some weird reason or bug.
I suspect some alias or function that adds (directly or through a variable) some options. Double-check that this isn't the case.
I have a root directory that I need to run a find and/or grep command on to return a list of files that contain a specific string.
Here's an example of the file and directory set up. In reality, this root directory contains a lot of subdirectories that each have a lot of subdirectories and files, but this example, I hope, gets my point across.
From root, I need to go through each of the children directories, specifically into subdir/ and look through file.html for the string "example:". If a result is found, I'd like it to print out the full path to file.html, such as website_two/subdir/file.html.
I figured limiting the search to subdir/file.html will greatly increase the speed of this operation.
I'm not too knowledgeable with find and grep commands, but I have tried the following with no luck, but I honestly don't know how to troubleshoot it.
find . -name "file.html" -exec grep -HI "example:" {} \;
EDIT: I understand this may be marked as a duplicate, but I think my question is more along the lines of how can I tell the command to only search a specific file in a specific path, looping through all root-> level directories.
find ./ -type f -iname file.html -exec grep -l "example:" {} \+;
or
grep -Rl "example:" ./ | grep -iE "file.htm(l)*$" will do the trick.
Quote from GNU Grep 2.25 man page:
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all symbolic links, unlike -r.
-l, --files-with-matches
Suppress normal output; instead print the name of each input file from which output would normally have
been printed. The scanning will stop on the first match.
-i, --ignore-case
Ignore case distinctions in both the PATTERN and the input files.
-E, --extended-regexp
Interpret PATTERN as an extended regular expression.
I tried to search the MarkDown files containing some specific text under the working directory, so I used the command below:
find . -name "*.md" | xargs grep "interpretation"
However, though I got the result what I need, the terminal also print out much errors like below:
grep: Problem: No such file or directory
grep: Solving: No such file or directory
grep: with: No such file or directory
grep: Algorithms: No such file or directory
……
etc
I wrote my solution as a answer below.
Found it!
At first I used the option -s to suppress errors, suggested by here, but #Kelvin 's comment reminded me the true reason is that my many files' names has the spaces.
So the correct command is:
$ find . -name "*.md" -print0 | xargs -0 grep "some-text-want-to-find" (on os x)
Here is some clearer explanation I found:
Unix-like systems allow embedded spaces (and even newlines!) in filenames. This causes problems for programs like xargs that construct argument lists for other programs. An embedded space will be treated as a delimiter and the resulting command will interpret each space-separated word as a separate argument. To overcome this, find and xarg allow the optional use of a null character as argument separator. A null character is defined in ASCII as the character represented by the number zero (as opposed to, for example, the space character, which is defined in ASCII as the character represented by the number 32). The find command provides the action -print0, which produces null separated output, and the xargs command has the –null option, which accepts null separated input.
—— The Linux Command Line: A Complete Introduction by William E. Shotts
Warning: When you are on os x, the null separated option of xargs command is –0
Updated 2017-05-27 22:58:48
Thanks to #Sundeep ,who suggested me to use -exec, a new feature in find itself, rather than xargs.
So, use this to search files in current dir and its sub-dirs:
$ find . -type f -name "*.md" -exec grep "some-text-want-to-find" {} +
Note:
What is meaning of {} + in find's -exec command? - Unix & Linux Stack Exchange
I found that when there's symbolic link and the destination directory not exist, it will print out no such file or directory
Ok what I am trying to do is very specific. I need some code that will remove files from a directory based on integer.name.
The files in the directory are listed like this
441.TERM (the # is actually a PID so it'll be random)
442.TERM
No matter what I always want to keep the first .TERM file & remove any .TERM file after that as no more than one should ever be created by my script, but it does happen sometimes due to some issues with the system I am scripting on. I only want it to effect my 000.TERM files any other files it finds in the directory can stay. So if directory contain any .TERM file an with an integer higher than the first one found then remove the .TERM files with higher integers.
PS. .TERM is not an extension just in case there is any confusion.
find /your/path -name "*.TERM" | sort -t. -k1 -n | tail -n +2 | xargs -r rm
Let's break it down:
find /your/path -name "*.TERM" will output a list of all .TERM files.
You could also use ls /your/path/*.TERM, but you may find the output unpredictable. (Example: your implementation may have -F on by default, which would cause every socket to end in a = in the list.)
sort sorts them by the first field (-k1) using a period as a separator (-t.). -n guarantees a numeric sort (such that 5 comes before 06).
tail -n +2 skips the first line and returns the rest
xargs rm sends every output line to an rm command, removing them. -r skips running the rm if there's no output piped in, but is listed as a GNU extension.
The script as above is fairly robust for your needs, but may fail if you have so many files in the directory that they don't fit on one command line, and might get you into trouble if any matching filenames somehow contain a newline.
If I do
$ ls -l --color=always
I get a list of files inside the directory with some nice colouring for different file types etc..
Now, I want to be able to pipe the coloured output of ls through grep to filter out some files I don't need. The key is that I still want to preserve the colouring after the grep filter.
$ ls -l --color=always | grep -E some_regex
^ I lose the colouring after grep
EDIT: I'm using headless-server Ubuntu 8.10, Bash 3.2.39, pretty much a stock install with no fancy configs
Your grep is probably removing ls's color codes because it has its own coloring turned on.
You "could" do this:
ls -l --color=always | grep --color=never pattern
However, it is very important that you understand what exactly you're grepping here. Not only is grepping ls unnecessary (use a glob instead), this particular case is grepping through not only filenames and file stats, but also through the color codes added by ls!
The real answer to your question is: Don't grep it. There is never a need to pipe ls into anything or capture its output. ls is only intended for human interpretation (eg. to look at in an interactive shell only, and for this purpose it is extremely handy, of course). As mentioned before, you can filter what files ls enumerates by using globs:
ls -l *.txt # Show all files with filenames ending with `.txt'.
ls -l !(foo).txt # Show all files with filenames that end on `.txt' but aren't `foo.txt'. (This requires `shopt -s extglob` to be on, you can put it in ~/.bashrc)
I highly recommend you read these two excellent documents on the matter:
Explanation of the badness of parsing ls: http://mywiki.wooledge.org/ParsingLs
The power of globs: http://mywiki.wooledge.org/glob
You should check if you are really using the "real" ls, just by directly calling the binary:
/bin/ls ....
Because: The code you described really should work, unless ls ignores --color=always for some weird reason or bug.
I suspect some alias or function that adds (directly or through a variable) some options. Double-check that this isn't the case.