I want to look through 100K+ text files from a directory and copy to another directory only the ones which contain at least one word from a list.
I tried doing an if statement with grep and cp but I have no idea how to make it to work this way.
for filename in *.txt
do
grep -o -i "cultiv" "protec" "agricult" $filename|wc -w
if [ wc -gt 0 ]
then cp $filename ~/desktop/filepath
fi
done
Obviously this does not work but I have no idea how to store the wc result and then compare it to 0 and only act on those files.
Use the -l option to have grep print all the filenames that match the pattern. Then use xargs to pass these as arguments to cp.
grep -l -E -i 'cultiv|protec|agricult' *.txt | xargs cp -t ~/desktop/filepath --
The -t option is a GNU cp extension, it allows you to put the destination directory first so that it will work with xargs.
If you're using a version without that option, you need to use the -J option to xargs to substitute in the middle of the command.
grep -l -E -i 'cultiv|protec|agricult' *.txt | xargs -J {} cp -- {} ~/desktop/filepath
I've spent hours searching and can't find a solution to this. I have a directory with over 1,000 PHP files. I need to replace some code in these files as follows:
Find:
session_register("CurWebsiteID");
Replace with (saved in replacement.txt:
if(!function_exists ("session_register") && isset($_SERVER["DOCUMENT_ROOT"])){require_once($_SERVER["DOCUMENT_ROOT"]."/libraries/phpruntime/php_legacy_session_functions.php");} session_register("CurWebsiteID");
Using the command below, I'm able to replace the pattern with $(cat replacement.txt) whereas I'm looking to replace them with the content of the text file.
Command being used:
find . -name "*.xml" | xargs -n 1 sed -i -e 's/mercy/$(cat replacement.txt)/g'
I've also tried using variables instead replacement=code_above; and running an adjusted version with $(echo $replacement) but that doesn't help either.
What is the correct way to achieve this?
You don't need command substitution here. You can use the sed r command to insert file content and d to delete the line matching the pattern:
find . -name "*.xml" | xargs -n 1 sed -i -e '/mercy/r replacement.txt' -e '//d'
$(...) is not interpreted inside single quotes. Use double quotes:
find . -name "*.xml" | xargs -n 1 sed -i -e "s/mercy/$(cat replacement.txt)/g"
You can also do away with cat:
find . -name "*.xml" | xargs -n 1 sed -i -e "s/mercy/$(< replacement.txt)/g"
In case replacement.txt has a / in it, use a different delimiter in sed expression, for example #:
find . -name "*.xml" | xargs -n 1 sed -i -e "s#mercy#$(< replacement.txt)#g"
See also:
Use slashes in sed replace
I want to change all occurrences of <ga/ to <. With xargs, this works fine:
ls | xargs sed -i 's/<ga\//</g'
GNU Parallel says that it's a direct replacement for xargs, but doing
ls | parallel sed -i 's/<ga\//</g'
results in
/bin/bash: ga//: No such file or directory
for each file in the directory. I'm sure that I'm just forgetting a {} or \; somewhere, but the answer still alludes me.
ls | parallel -q sed -i 's/<ga\//</g'
I have tried this command,
grep '/static' dir/* | xargs sed -i 's/\/static//g'
but the version of sed I am using does not support the -i argument.
To replace a string in a file, to the same input file as the output, I normally do this:
sed 's/\/static//g' filename.txt > new_filename.txt ; mv new_filename.txt filename.txt
OS X's version of sed does support -i, but it requires an argument to tell it what file extension to use for the backup file (or "" for no backup). BTW, you want grep -l to get just the filenames.
grep -l '/static' dir/* | xargs sed -i "" 's/\/static//g'
Use perl:
$ perl -pi.bak -e 's#/static##g' dir/*
You can do this using a loop:
for file in $(grep -l '/static' dir/*) ; do
sed 's/\/static//g' $file > $file.$$ && mv $file.$$ $file
done
I use the .$$ suffix ($$ is the process id of the current shell) to avoid collisions with existing file names, and && rather than ; to avoid clobbering the input file if the sed command fails for some reason. I also added -l so grep prints file names rather than matching lines.
Or you can install GNU sed (I'm not sure exactly how to do that on OSX).
what's the best way to do this? I'm no command line warrior, but I was thinking there's possibly a way of using grep and cat.
I just want to replace a string that occurs in a folder and sub-folders. what's the best way to do this? I'm running ubuntu if that matters.
I'll throw in another example for folks using ag, The Silver Searcher to do find/replace operations on multiple files.
Complete example:
ag -l "search string" | xargs sed -i '' -e 's/from/to/g'
If we break this down, what we get is:
# returns a list of files containing matching string
ag -l "search string"
Next, we have:
# consume the list of piped files and prepare to run foregoing command
# for each file delimited by newline
xargs
Finally, the string replacement command:
# -i '' means edit files in place and the '' means do not create a backup
# -e 's/from/to/g' specifies the command to run, in this case,
# global, search and replace
sed -i '' -e 's/from/to/g'
find . -type f -print0 | xargs -0 -n 1 sed -i -e 's/from/to/g'
The first part of that is a find command to find the files you want to change. You may need to modify that appropriately. The xargs command takes every file the find found and applies the sed command to it. The sed command takes every instance of from and replaces it with to. That's a standard regular expression, so modify it as you need.
If you are using svn beware. Your .svn-directories will be search and replaced as well. You have to exclude those, e.g., like this:
find . ! -regex ".*[/]\.svn[/]?.*" -type f -print0 | xargs -0 -n 1 sed -i -e 's/from/to/g'
or
find . -name .svn -prune -o -type f -print0 | xargs -0 -n 1 sed -i -e 's/from/to/g'
As Paul said, you want to first find the files you want to edit and then edit them. An alternative to using find is to use GNU grep (the default on Ubuntu), e.g.:
grep -r -l from . | xargs -0 -n 1 sed -i -e 's/from/to/g'
You can also use ack-grep (sudo apt-get install ack-grep or visit http://petdance.com/ack/) as well, if you know you only want a certain type of file, and want to ignore things in version control directories. e.g., if you only want text files,
ack -l --print0 --text from | xargs -0 -n 1 sed -i -e 's/from/to/g'
# `from` here is an arbitrary commonly occurring keyword
An alternative to using sed is to use perl which can process multiple files per command, e.g.,
grep -r -l from . | xargs perl -pi.bak -e 's/from/to/g'
Here, perl is told to edit in place, making a .bak file first.
You can combine any of the left-hand sides of the pipe with the right-hand sides, depending on your preference.
An alternative to sed is using rpl (e.g. available from http://rpl.sourceforge.net/ or your GNU/Linux distribution), like rpl --recursive --verbose --whole-words 'F' 'A' grades/
For convenience, I took Ulysse's answer (after correcting the undesirable error printing) and turned it into a .zshrc / .bashrc function:
function find-and-replace() {
ag -l "$1" | xargs sed -i -e s/"$1"/"$2"/g
}
Usage: find-and-replace Foo Bar
The typical (find|grep|ack|ag|rg)-xargs-sed combination has a few problems:
Difficult to remember and get correct. Eg, forgetting the xargs -r option will run the command even when no files are found, potentially causing problems.
Retrieving the file list, and the actual replacement uses different CLI tools and can have a different search behaviour.
These problems were big enough for such an invasive and dangerous operation as recursive search-and-replace, to start the development of a dedicated tool: mo.
Early tests seem to indicate that its performance is between ag and rg and it solves following problems I encounter with them:
A single invocation can filter on filename and content. Following command searches for the word bug in all source files that have a v1 indication:
mo -f 'src/.*v1.*' -p bug -w
Once the search results are OK, actual replacement for bug with fix can be added:
mo -f 'src/.*v1.*' -p bug -w -r fix
comment() {
}
doc() {
}
function agr {
doc 'usage: from=sth to=another agr [ag-args]'
comment -l --files-with-matches
ag -0 -l "$from" "${#}" | pre-files "$from" "$to"
}
pre-files() {
doc 'stdin should be null-separated list of files that need replacement; $1 the string to replace, $2 the replacement.'
comment '-i backs up original input files with the supplied extension (leave empty for no backup; needed for in-place replacement.)(do not put whitespace between -i and its arg.)'
comment '-r, --no-run-if-empty
If the standard input does not contain any nonblanks,
do not run the command. Normally, the command is run
once even if there is no input. This option is a GNU
extension.'
AGR_FROM="$1" AGR_TO="$2" xargs -r0 perl -pi.pbak -e 's/$ENV{AGR_FROM}/$ENV{AGR_TO}/g'
}
You can use it like this:
from=str1 to=sth agr path1 path2 ...
Supply no paths to make it use the current directory.
Note that ag, xargs, and perl need to be installed and on PATH.