Sed with Xargs cannot open passed file (Cygwin) - windows

Trying to use the beauty of Sed so I don't have to manually update a few hundred files. I'll note my employer only allows use of Win8 (joy), so I use Cygwin all day until I can use my Linux boxes at home.
The following works on a Linux (bash) command line, but not Cygwin
> grep -lrZ "/somefile.js" . | xargs -0 -l sed -i -e 's|/somefile.js|/newLib.js|g'
sed: can't read ./testTarget.jsp: No such file or directory
# works
> sed -i 's|/somefile.js|/newLib.js|g' ./testTarget.jsp
So the command by itself works, but not passed through Xargs. And, before you say to use Perl instead of Sed, the Perl equivalent throws the very same error
> grep -lrZ "/somefile.js" . | xargs -0 perl -i -pe 's|/somefile.js|/newLib.js|g'
Can't open ./testTarget.jsp
: No such file or directory.

Use the xargs -n option to split up the arguments and force separate calls to sed.
On windows using GnuWin tools (not Cygwin) I found that I need to split up the input to sed. By default xargs will pass ALL of the files from grep to one call to sed.
Let's say you have 4 files that match your grep call, the sed command will run through xargs like this:
sed -i -e 's|/somefile.js|/newLib.js|g' ./file1 ./file2 ./subdir/file3 ./subdir/file4
If the number of files is too large sed will give you this error.
Use the -n option to have xargs call sed repeatedly until it exhausts all of the arguments.
grep -lrZ "/somefile.js" . | xargs -0 -l -n 2 sed -i -e 's|/somefile.js|/newLib.js|g'
In my small example using -n 2 will internally do this:
sed -i -e 's|/somefile.js|/newLib.js|g' ./file1 ./file2
sed -i -e 's|/somefile.js|/newLib.js|g' ./subdir/file3 ./subdir/file4
I had a large set of files and directories (around 3000 files), and using xargs -n 5 worked great.
When I tried -n 10 I got errors. Using xargs --verbose I could see some of the commandline calls were getting cut off at around 500 characters. So you may need to make -n smaller depending on the path length of the files you are woking with.

Related

Copy files that have at least the mention of one certain word

I want to look through 100K+ text files from a directory and copy to another directory only the ones which contain at least one word from a list.
I tried doing an if statement with grep and cp but I have no idea how to make it to work this way.
for filename in *.txt
do
grep -o -i "cultiv" "protec" "agricult" $filename|wc -w
if [ wc -gt 0 ]
then cp $filename ~/desktop/filepath
fi
done
Obviously this does not work but I have no idea how to store the wc result and then compare it to 0 and only act on those files.
Use the -l option to have grep print all the filenames that match the pattern. Then use xargs to pass these as arguments to cp.
grep -l -E -i 'cultiv|protec|agricult' *.txt | xargs cp -t ~/desktop/filepath --
The -t option is a GNU cp extension, it allows you to put the destination directory first so that it will work with xargs.
If you're using a version without that option, you need to use the -J option to xargs to substitute in the middle of the command.
grep -l -E -i 'cultiv|protec|agricult' *.txt | xargs -J {} cp -- {} ~/desktop/filepath

Using a file's content in sed's replacement string

I've spent hours searching and can't find a solution to this. I have a directory with over 1,000 PHP files. I need to replace some code in these files as follows:
Find:
session_register("CurWebsiteID");
Replace with (saved in replacement.txt:
if(!function_exists ("session_register") && isset($_SERVER["DOCUMENT_ROOT"])){require_once($_SERVER["DOCUMENT_ROOT"]."/libraries/phpruntime/php_legacy_session_functions.php");} session_register("CurWebsiteID");
Using the command below, I'm able to replace the pattern with $(cat replacement.txt) whereas I'm looking to replace them with the content of the text file.
Command being used:
find . -name "*.xml" | xargs -n 1 sed -i -e 's/mercy/$(cat replacement.txt)/g'
I've also tried using variables instead replacement=code_above; and running an adjusted version with $(echo $replacement) but that doesn't help either.
What is the correct way to achieve this?
You don't need command substitution here. You can use the sed r command to insert file content and d to delete the line matching the pattern:
find . -name "*.xml" | xargs -n 1 sed -i -e '/mercy/r replacement.txt' -e '//d'
$(...) is not interpreted inside single quotes. Use double quotes:
find . -name "*.xml" | xargs -n 1 sed -i -e "s/mercy/$(cat replacement.txt)/g"
You can also do away with cat:
find . -name "*.xml" | xargs -n 1 sed -i -e "s/mercy/$(< replacement.txt)/g"
In case replacement.txt has a / in it, use a different delimiter in sed expression, for example #:
find . -name "*.xml" | xargs -n 1 sed -i -e "s#mercy#$(< replacement.txt)#g"
See also:
Use slashes in sed replace

GNU Parallel with sed wrong arguement as file

I want to change all occurrences of <ga/ to <. With xargs, this works fine:
ls | xargs sed -i 's/<ga\//</g'
GNU Parallel says that it's a direct replacement for xargs, but doing
ls | parallel sed -i 's/<ga\//</g'
results in
/bin/bash: ga//: No such file or directory
for each file in the directory. I'm sure that I'm just forgetting a {} or \; somewhere, but the answer still alludes me.
ls | parallel -q sed -i 's/<ga\//</g'

How to replace a string in multiple files using grep and sed when the -i argument is not supported?

I have tried this command,
grep '/static' dir/* | xargs sed -i 's/\/static//g'
but the version of sed I am using does not support the -i argument.
To replace a string in a file, to the same input file as the output, I normally do this:
sed 's/\/static//g' filename.txt > new_filename.txt ; mv new_filename.txt filename.txt
OS X's version of sed does support -i, but it requires an argument to tell it what file extension to use for the backup file (or "" for no backup). BTW, you want grep -l to get just the filenames.
grep -l '/static' dir/* | xargs sed -i "" 's/\/static//g'
Use perl:
$ perl -pi.bak -e 's#/static##g' dir/*
You can do this using a loop:
for file in $(grep -l '/static' dir/*) ; do
sed 's/\/static//g' $file > $file.$$ && mv $file.$$ $file
done
I use the .$$ suffix ($$ is the process id of the current shell) to avoid collisions with existing file names, and && rather than ; to avoid clobbering the input file if the sed command fails for some reason. I also added -l so grep prints file names rather than matching lines.
Or you can install GNU sed (I'm not sure exactly how to do that on OSX).

Best way to do a find/replace in several files?

what's the best way to do this? I'm no command line warrior, but I was thinking there's possibly a way of using grep and cat.
I just want to replace a string that occurs in a folder and sub-folders. what's the best way to do this? I'm running ubuntu if that matters.
I'll throw in another example for folks using ag, The Silver Searcher to do find/replace operations on multiple files.
Complete example:
ag -l "search string" | xargs sed -i '' -e 's/from/to/g'
If we break this down, what we get is:
# returns a list of files containing matching string
ag -l "search string"
Next, we have:
# consume the list of piped files and prepare to run foregoing command
# for each file delimited by newline
xargs
Finally, the string replacement command:
# -i '' means edit files in place and the '' means do not create a backup
# -e 's/from/to/g' specifies the command to run, in this case,
# global, search and replace
sed -i '' -e 's/from/to/g'
find . -type f -print0 | xargs -0 -n 1 sed -i -e 's/from/to/g'
The first part of that is a find command to find the files you want to change. You may need to modify that appropriately. The xargs command takes every file the find found and applies the sed command to it. The sed command takes every instance of from and replaces it with to. That's a standard regular expression, so modify it as you need.
If you are using svn beware. Your .svn-directories will be search and replaced as well. You have to exclude those, e.g., like this:
find . ! -regex ".*[/]\.svn[/]?.*" -type f -print0 | xargs -0 -n 1 sed -i -e 's/from/to/g'
or
find . -name .svn -prune -o -type f -print0 | xargs -0 -n 1 sed -i -e 's/from/to/g'
As Paul said, you want to first find the files you want to edit and then edit them. An alternative to using find is to use GNU grep (the default on Ubuntu), e.g.:
grep -r -l from . | xargs -0 -n 1 sed -i -e 's/from/to/g'
You can also use ack-grep (sudo apt-get install ack-grep or visit http://petdance.com/ack/) as well, if you know you only want a certain type of file, and want to ignore things in version control directories. e.g., if you only want text files,
ack -l --print0 --text from | xargs -0 -n 1 sed -i -e 's/from/to/g'
# `from` here is an arbitrary commonly occurring keyword
An alternative to using sed is to use perl which can process multiple files per command, e.g.,
grep -r -l from . | xargs perl -pi.bak -e 's/from/to/g'
Here, perl is told to edit in place, making a .bak file first.
You can combine any of the left-hand sides of the pipe with the right-hand sides, depending on your preference.
An alternative to sed is using rpl (e.g. available from http://rpl.sourceforge.net/ or your GNU/Linux distribution), like rpl --recursive --verbose --whole-words 'F' 'A' grades/
For convenience, I took Ulysse's answer (after correcting the undesirable error printing) and turned it into a .zshrc / .bashrc function:
function find-and-replace() {
ag -l "$1" | xargs sed -i -e s/"$1"/"$2"/g
}
Usage: find-and-replace Foo Bar
The typical (find|grep|ack|ag|rg)-xargs-sed combination has a few problems:
Difficult to remember and get correct. Eg, forgetting the xargs -r option will run the command even when no files are found, potentially causing problems.
Retrieving the file list, and the actual replacement uses different CLI tools and can have a different search behaviour.
These problems were big enough for such an invasive and dangerous operation as recursive search-and-replace, to start the development of a dedicated tool: mo.
Early tests seem to indicate that its performance is between ag and rg and it solves following problems I encounter with them:
A single invocation can filter on filename and content. Following command searches for the word bug in all source files that have a v1 indication:
mo -f 'src/.*v1.*' -p bug -w
Once the search results are OK, actual replacement for bug with fix can be added:
mo -f 'src/.*v1.*' -p bug -w -r fix
comment() {
}
doc() {
}
function agr {
doc 'usage: from=sth to=another agr [ag-args]'
comment -l --files-with-matches
ag -0 -l "$from" "${#}" | pre-files "$from" "$to"
}
pre-files() {
doc 'stdin should be null-separated list of files that need replacement; $1 the string to replace, $2 the replacement.'
comment '-i backs up original input files with the supplied extension (leave empty for no backup; needed for in-place replacement.)(do not put whitespace between -i and its arg.)'
comment '-r, --no-run-if-empty
If the standard input does not contain any nonblanks,
do not run the command. Normally, the command is run
once even if there is no input. This option is a GNU
extension.'
AGR_FROM="$1" AGR_TO="$2" xargs -r0 perl -pi.pbak -e 's/$ENV{AGR_FROM}/$ENV{AGR_TO}/g'
}
You can use it like this:
from=str1 to=sth agr path1 path2 ...
Supply no paths to make it use the current directory.
Note that ag, xargs, and perl need to be installed and on PATH.

Resources