Related
Saying that I have two files t1 and t2, they have the same content: abc.
Now I want to delete all files, who contains the string abc.
So I tried to execute the command: grep -rl abc . | rm but it doesn't work.
Then I add xargs: grep -rl abc . | xargs rm and it works.
I can't understand clearly what xargs did.
grep puts the output as stdout. But rm cannot process data from stdin (the pipe links both).
You want instead, that the output of grep is put as argument of rm. So xargs command "convert" stdin into arguments of xargs first argument, and it call the command (the first argument).
As alternative, you could do
rm `grep -rl abc .`
or
rm $(grep -rl abc .)
But xargs handles well also the case where there are too many arguments for a single call of the command. The above command will give you shell error (argument string too long).
rm doesn't read from standard input (except when prompting, like with -i) but takes its arguments on the command line. That's what xargs does for you: read things from standard input and give them to rm as arguments.
Example with echo:
$ (echo a; echo b; date) | xargs echo
a b tor 12 apr 2018 14:18:50 CEST
I am using jsonlint to lint a bunch of files in a directory (recursively). I wrote the following command:
find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'echo Linting: %; jsonlint -V ./config/schema.json -q %;'
It works for most files but some files I get the following error:
Linting: ./LONG_FILE_NAME.json
fs.js:500
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Error: ENOENT, no such file or directory '%'
It appears to fail for long filenames. Is there a way to fix this? Thanks.
Edit 1:
Found the problem.
-I replstr
Execute utility for each input line, replacing one or more occurrences
of replstr in up to replacements (or 5 if no -R flag is specified)
arguments to utility with the entire line of input. The resulting
arguments, after replacement is done, will not be allowed to grow
beyond 255 bytes; this is implemented by concatenating as much of the
argument containing replstr as possible, to the con-structed arguments
to utility, up to 255 bytes. The 255 byte limit does not apply to
arguments to utility which do not contain replstr, and furthermore, no
replacement will be done on utility itself. Implies -x.
Edit 2:
Partial solution. Supports longer file names than before but still not as long as I need.
find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'file=%; echo Linting: $file; jsonlint -V ./config/schema.json -q $file;'
On BSD like systems (e.g. Mac OS X)
If you happen to be on a mac or freebsd etc. your xargs implementation may support option -J which does not suffer from the argument size limits imposed on option -I.
Excert from manpage
-J replstr
If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not effect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following command will copy the list of files and directories which start with an uppercase letter in the current directory to destdir:
/bin/ls -1d [A-Z]* | xargs -J % cp -Rp % destdir
If you need to refer to the repstr multiple times (*points up* TL;DR -J only replaces first occurrence) you can use this pattern:
echo hi | xargs -J{} sh -c 'arg=$0; echo "$arg $arg"' "{}"
=> hi hi
POSIX compliant method
The posix compliant method of doing this would be to use some other tool, e.g. sed to construct the code you want to execute and then use xargs to just specify the utility. When no repl string is used in xargs the 255 byte limit does not apply. xargs POSIX spec
find . -type f -name '*.json' -print |
sed "s_^_-c 'file=\\\"_g;s_\$_\\\"; echo \\\"Definitely over 255 byte script..$(printf "a%.0s" {1..255}): \\\$file\\\"; wc -l \\\"\\\$file\\\"'_g" |
xargs -L1 sh
This of course largely defeats the purpose of xargs to begin with, but can still be used to leverage e.g. parallel execution using xargs -L1 -P10 sh which is quite widely supported, though not posix.
Use -exec in find instead of piping to xargs.
find ./config/pages -name '*.json' -print0 -exec echo Linting: {} \; -exec jsonlint -V ./config/schema.json -q {} \;
The limit on xargs's command line length is imposed by the system (not an environment) variable ARG_MAX. You can check it like:
$ getconf ARG_MAX
2097152
Surprisingly, there doesn't not seem to be a way to change it, barring kernel modification.
But even more surprising that xargs by default gets capped to a much lower value, and you can increase with -s option. Still, ARG_MAX is not the value you can set after -s — acc. to man xargs you need to subtract size of environment, plus some "headroom", no idea why. To find out the actual number use the following command (alternatively, using an arbitrary big number for -s will result in a descriptive error):
$ xargs --show-limits 2>&1 | grep "limit on argument length (this system)"
POSIX upper limit on argument length (this system): 2092120
So you need to run … | xargs -s 2092120 …, e.g. with your command:
find ./config/pages -name '*.json' -print0 | xargs -s 2092120 -0I % sh -c 'echo Linting: %; jsonlint -V ./config/schema.json -q %;'
How do I pipe commands and their results in Ubuntu when writing them in the terminal. I would write the following commands in sequence -
$ ls | grep ab
abc.pdf
cde.pdf
$ cp abc.pdf cde.pdf files/
I would like to pipe the results of the first command into the second command, and write them all in the same line. How do I do that ?
something like
$ cp "ls | grep ab" files/
(the above is a contrived example and can be written as cp *.pdf files/)
Use the following:
cp `ls | grep ab` files/
Well, since the xargs person gave up, I'll offer my xargs solution:
ls | grep ab | xargs echo | while read f; do cp $f files/; done
Of course, this solution suffers from an obvious flaw: files with spaces in them will cause chaos.
An xargs solution without this flaw? Hmm...
ls | grep ab | xargs '-d\n' bash -c 'docp() { cp "$#" files/; }; docp "$#"'
Seems a bit klunky, but it works. Unless you have files with returns in them I mean. However, anyone who does that deserves what they get. Even that is solvable:
find . -mindepth 1 -maxdepth 1 -name '*ab*' -print0 | xargs -0 bash -c 'docp() { cp "$#" files/; }; docp "$#"'
To use xargs, you need to ensure that the filename arguments are the last arguments passed to the cp command. You can accomplish this with the -t option to cp to specify the target directory:
ls | grep ab | xargs cp -t files/
Of course, even though this is a contrived example, you should not parse the output of ls.
How can I make xargs execute the command exactly once for each line of input given?
It's default behavior is to chunk the lines and execute the command once, passing multiple lines to each instance.
From http://en.wikipedia.org/wiki/Xargs:
find /path -type f -print0 | xargs -0 rm
In this example, find feeds the input of xargs with a long list of file names. xargs then splits this list into sublists and calls rm once for every sublist. This is more efficient than this functionally equivalent version:
find /path -type f -exec rm '{}' \;
I know that find has the "exec" flag. I am just quoting an illustrative example from another resource.
The following will only work if you do not have spaces in your input:
xargs -L 1
xargs --max-lines=1 # synonym for the -L option
from the man page:
-L max-lines
Use at most max-lines nonblank input lines per command line.
Trailing blanks cause an input line to be logically continued on
the next input line. Implies -x.
It seems to me all existing answers on this page are wrong, including the one marked as correct. That stems from the fact that the question is ambiguously worded.
Summary: If you want to execute the command "exactly once for each line of input," passing the entire line (without newline) to the command as a single argument, then this is the best UNIX-compatible way to do it:
... | tr '\n' '\0' | xargs -0 -n1 ...
If you are using GNU xargs and don't need to be compatible with all other UNIX's (FreeBSD, Mac OS X, etc.) then you can use the GNU-specific option -d:
... | xargs -d\\n -n1 ...
Now for the long explanation…
There are two issues to take into account when using xargs:
how does it split the input into "arguments"; and
how many arguments to pass the child command at a time.
To test xargs' behavior, we need an utility that shows how many times it's being executed and with how many arguments. I don't know if there is a standard utility to do that, but we can code it quite easily in bash:
#!/bin/bash
echo -n "-> "; for a in "$#"; do echo -n "\"$a\" "; done; echo
Assuming you save it as show in your current directory and make it executable, here is how it works:
$ ./show one two 'three and four'
-> "one" "two" "three and four"
Now, if the original question is really about point 2. above (as I think it is, after reading it a few times over) and it is to be read like this (changes in bold):
How can I make xargs execute the command exactly once for each argument of input given? Its default behavior is to chunk the input into arguments and execute the command as few times as possible, passing multiple arguments to each instance.
then the answer is -n 1.
Let's compare xargs' default behavior, which splits the input around whitespace and calls the command as few times as possible:
$ echo one two 'three and four' | xargs ./show
-> "one" "two" "three" "and" "four"
and its behavior with -n 1:
$ echo one two 'three and four' | xargs -n 1 ./show
-> "one"
-> "two"
-> "three"
-> "and"
-> "four"
If, on the other hand, the original question was about point 1. input splitting and it was to be read like this (many people coming here seem to think that's the case, or are confusing the two issues):
How can I make xargs execute the command with exactly one argument for each line of input given? Its default behavior is to chunk the lines around whitespace.
then the answer is more subtle.
One would think that -L 1 could be of help, but it turns out it doesn't change argument parsing. It only executes the command once for each input line, with as many arguments as were there on that input line:
$ echo $'one\ntwo\nthree and four' | xargs -L 1 ./show
-> "one"
-> "two"
-> "three" "and" "four"
Not only that, but if a line ends with whitespace, it is appended to the next:
$ echo $'one \ntwo\nthree and four' | xargs -L 1 ./show
-> "one" "two"
-> "three" "and" "four"
Clearly, -L is not about changing the way xargs splits the input into arguments.
The only argument that does so in a cross-platform fashion (excluding GNU extensions) is -0, which splits the input around NUL bytes.
Then, it's just a matter of translating newlines to NUL with the help of tr:
$ echo $'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 ./show
-> "one " "two" "three and four"
Now the argument parsing looks all right, including the trailing whitespace.
Finally, if you combine this technique with -n 1, you get exactly one command execution per input line, whatever input you have, which may be yet another way to look at the original question (possibly the most intuitive, given the title):
$ echo $'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 -n1 ./show
-> "one "
-> "two"
-> "three and four"
As mentioned above, if you are using GNU xargs you can replace the tr with the GNU-specific option -d:
$ echo $'one \ntwo\nthree and four' | xargs -d\\n -n1 ./show
-> "one "
-> "two"
-> "three and four"
If you want to run the command for every line (i.e. result) coming from find, then what do you need the xargs for?
Try:
find path -type f -exec your-command {} \;
where the literal {} gets substituted by the filename and the literal \; is needed for find to know that the custom command ends there.
EDIT:
(after the edit of your question clarifying that you know about -exec)
From man xargs:
-L max-lines
Use at most max-lines nonblank input lines per command line. Trailing
blanks cause an input line to be logically continued on the next input line.
Implies -x.
Note that filenames ending in blanks would cause you trouble if you use xargs:
$ mkdir /tmp/bax; cd /tmp/bax
$ touch a\ b c\ c
$ find . -type f -print | xargs -L1 wc -l
0 ./c
0 ./c
0 total
0 ./b
wc: ./a: No such file or directory
So if you don't care about the -exec option, you better use -print0 and -0:
$ find . -type f -print0 | xargs -0L1 wc -l
0 ./c
0 ./c
0 ./b
0 ./a
How can I make xargs execute the command exactly once for each line of input given?
-L 1 is the simple solution but it does not work if any of the files contain spaces in them. This is a key function of find's -print0 argument – to separate the arguments by '\0' character instead of whitespace. Here's an example:
echo "file with space.txt" | xargs -L 1 ls
ls: file: No such file or directory
ls: with: No such file or directory
ls: space.txt: No such file or directory
A better solution is to use tr to convert newlines to null (\0) characters, and then use the xargs -0 argument. Here's an example:
echo "file with space.txt" | tr '\n' '\0' | xargs -0 ls
file with space.txt
If you then need to limit the number of calls you can use the -n 1 argument to make one call to the program for each input:
echo "file with space.txt" | tr '\n' '\0' | xargs -0 -n 1 ls
This also allows you to filter the output of find before converting the breaks into nulls.
find . -name \*.xml | grep -v /target/ | tr '\n' '\0' | xargs -0 tar -cf xml.tar
These two ways also work, and will work for other commands that are not using find!
xargs -I '{}' rm '{}'
xargs -i rm '{}'
example use case:
find . -name "*.pyc" | xargs -i rm '{}'
will delete all pyc files under this directory even if the pyc files contain spaces.
Another alternative...
find /path -type f | while read ln; do echo "processing $ln"; done
find path -type f | xargs -L1 command
is all you need.
The following command will find all the files (-type f) in /path and then copy them using cp to the current folder. Note the use if -I % to specify a placeholder character in the cp command line so that arguments can be placed after the file name.
find /path -type f -print0 | xargs -0 -I % cp % .
Tested with xargs (GNU findutils) 4.4.0
You can limit the number of lines, or arguments (if there are spaces between each argument) using the --max-lines or --max-args flags, respectively.
-L max-lines
Use at most max-lines nonblank input lines per command line. Trailing blanks cause an input line to be logically continued on the next input
line. Implies -x.
--max-lines[=max-lines], -l[max-lines]
Synonym for the -L option. Unlike -L, the max-lines argument is optional. If max-args is not specified, it defaults to one. The -l option
is deprecated since the POSIX standard specifies -L instead.
--max-args=max-args, -n max-args
Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded,
unless the -x option is given, in which case xargs will exit.
#Draemon answers seems to be right with "-0" even with space in the file.
I was trying the xargs command and I found that "-0" works perfectly with "-L". even the spaces are treated (if input was null terminated ). the following is an example :
#touch "file with space"
#touch "file1"
#touch "file2"
The following will split the nulls and execute the command on each argument in the list :
#find . -name 'file*' -print0 | xargs -0 -L1
./file with space
./file1
./file2
so -L1 will execute the argument on each null terminated character if used with "-0". To see the difference try :
#find . -name 'file*' -print0 | xargs -0 | xargs -L1
./file with space ./file1 ./file2
even this will execute once :
#find . -name 'file*' -print0 | xargs -0 | xargs -0 -L1
./file with space ./file1 ./file2
The command will execute once as the "-L" now doesn't split on null byte. you need to provide both "-0" and "-L" to work.
It seems I don't have enough reputation to add a comment to Tobia's answer above, so I am adding this "answer" to help those of us wanting to experiment with xargs the same way on the Windows platforms.
Here is a windows batch file that does the same thing as Tobia's quickly coded "show" script:
#echo off
REM
REM cool trick of using "set" to echo without new line
REM (from: http://www.psteiner.com/2012/05/windows-batch-echo-without-new-line.html)
REM
if "%~1" == "" (
exit /b
)
<nul set /p=Args: "%~1"
shift
:start
if not "%~1" == "" (
<nul set /p=, "%~1"
shift
goto start
)
echo.
In your example, the point of piping the output of find to xargs is that the standard behavior of find's -exec option is to execute the command once for each found file. If you're using find, and you want its standard behavior, then the answer is simple - don't use xargs to begin with.
execute ant task clean-all on every build.xml on current or sub-folder.
find . -name 'build.xml' -exec ant -f {} clean-all \;
Is there a simple way, in a pretty standard UNIX environment with bash, to run a command to delete all but the most recent X files from a directory?
To give a bit more of a concrete example, imagine some cron job writing out a file (say, a log file or a tar-ed up backup) to a directory every hour. I'd like a way to have another cron job running which would remove the oldest files in that directory until there are less than, say, 5.
And just to be clear, there's only one file present, it should never be deleted.
The problems with the existing answers:
inability to handle filenames with embedded spaces or newlines.
in the case of solutions that invoke rm directly on an unquoted command substitution (rm `...`), there's an added risk of unintended globbing.
inability to distinguish between files and directories (i.e., if directories happened to be among the 5 most recently modified filesystem items, you'd effectively retain fewer than 5 files, and applying rm to directories will fail).
wnoise's answer addresses these issues, but the solution is GNU-specific (and quite complex).
Here's a pragmatic, POSIX-compliant solution that comes with only one caveat: it cannot handle filenames with embedded newlines - but I don't consider that a real-world concern for most people.
For the record, here's the explanation for why it's generally not a good idea to parse ls output: http://mywiki.wooledge.org/ParsingLs
ls -tp | grep -v '/$' | tail -n +6 | xargs -I {} rm -- {}
Note: This command operates in the current directory; to target a directory explicitly, use a subshell ((...)) with cd:
(cd /path/to && ls -tp | grep -v '/$' | tail -n +6 | xargs -I {} rm -- {})
The same applies analogously to the commands below.
The above is inefficient, because xargs has to invoke rm separately for each filename.
However, your platform's specific xargs implementation may allow you to solve this problem:
A solution that works with GNU xargs is to use -d '\n', which makes xargs consider each input line a separate argument, yet passes as many arguments as will fit on a command line at once:
ls -tp | grep -v '/$' | tail -n +6 | xargs -d '\n' -r rm --
Note: Option -r (--no-run-if-empty) ensures that rm is not invoked if there's no input.
A solution that works with both GNU xargs and BSD xargs (including on macOS) - though technically still not POSIX-compliant - is to use -0 to handle NUL-separated input, after first translating newlines to NUL (0x0) chars., which also passes (typically) all filenames at once:
ls -tp | grep -v '/$' | tail -n +6 | tr '\n' '\0' | xargs -0 rm --
Explanation:
ls -tp prints the names of filesystem items sorted by how recently they were modified , in descending order (most recently modified items first) (-t), with directories printed with a trailing / to mark them as such (-p).
Note: It is the fact that ls -tp always outputs file / directory names only, not full paths, that necessitates the subshell approach mentioned above for targeting a directory other than the current one ((cd /path/to && ls -tp ...)).
grep -v '/$' then weeds out directories from the resulting listing, by omitting (-v) lines that have a trailing / (/$).
Caveat: Since a symlink that points to a directory is technically not itself a directory, such symlinks will not be excluded.
tail -n +6 skips the first 5 entries in the listing, in effect returning all but the 5 most recently modified files, if any.
Note that in order to exclude N files, N+1 must be passed to tail -n +.
xargs -I {} rm -- {} (and its variations) then invokes on rm on all these files; if there are no matches at all, xargs won't do anything.
xargs -I {} rm -- {} defines placeholder {} that represents each input line as a whole, so rm is then invoked once for each input line, but with filenames with embedded spaces handled correctly.
-- in all cases ensures that any filenames that happen to start with - aren't mistaken for options by rm.
A variation on the original problem, in case the matching files need to be processed individually or collected in a shell array:
# One by one, in a shell loop (POSIX-compliant):
ls -tp | grep -v '/$' | tail -n +6 | while IFS= read -r f; do echo "$f"; done
# One by one, but using a Bash process substitution (<(...),
# so that the variables inside the `while` loop remain in scope:
while IFS= read -r f; do echo "$f"; done < <(ls -tp | grep -v '/$' | tail -n +6)
# Collecting the matches in a Bash *array*:
IFS=$'\n' read -d '' -ra files < <(ls -tp | grep -v '/$' | tail -n +6)
printf '%s\n' "${files[#]}" # print array elements
Remove all but 5 (or whatever number) of the most recent files in a directory.
rm `ls -t | awk 'NR>5'`
(ls -t|head -n 5;ls)|sort|uniq -u|xargs rm
This version supports names with spaces:
(ls -t|head -n 5;ls)|sort|uniq -u|sed -e 's,.*,"&",g'|xargs rm
Simpler variant of thelsdj's answer:
ls -tr | head -n -5 | xargs --no-run-if-empty rm
ls -tr displays all the files, oldest first (-t newest first, -r reverse).
head -n -5 displays all but the 5 last lines (ie the 5 newest files).
xargs rm calls rm for each selected file.
find . -maxdepth 1 -type f -printf '%T# %p\0' | sort -r -z -n | awk 'BEGIN { RS="\0"; ORS="\0"; FS="" } NR > 5 { sub("^[0-9]*(.[0-9]*)? ", ""); print }' | xargs -0 rm -f
Requires GNU find for -printf, and GNU sort for -z, and GNU awk for "\0", and GNU xargs for -0, but handles files with embedded newlines or spaces.
All these answers fail when there are directories in the current directory. Here's something that works:
find . -maxdepth 1 -type f | xargs -x ls -t | awk 'NR>5' | xargs -L1 rm
This:
works when there are directories in the current directory
tries to remove each file even if the previous one couldn't be removed (due to permissions, etc.)
fails safe when the number of files in the current directory is excessive and xargs would normally screw you over (the -x)
doesn't cater for spaces in filenames (perhaps you're using the wrong OS?)
ls -tQ | tail -n+4 | xargs rm
List filenames by modification time, quoting each filename. Exclude first 3 (3 most recent). Remove remaining.
EDIT after helpful comment from mklement0 (thanks!): corrected -n+3 argument, and note this will not work as expected if filenames contain newlines and/or the directory contains subdirectories.
Ignoring newlines is ignoring security and good coding. wnoise had the only good answer. Here is a variation on his that puts the filenames in an array $x
while IFS= read -rd ''; do
x+=("${REPLY#* }");
done < <(find . -maxdepth 1 -printf '%T# %p\0' | sort -r -z -n )
For Linux (GNU tools), an efficient & robust way to keep the n newest files in the current directory while removing the rest:
n=5
find . -maxdepth 1 -type f -printf '%T# %p\0' |
sort -z -nrt ' ' -k1,1 |
sed -z -e "1,${n}d" -e 's/[^ ]* //' |
xargs -0r rm -f
For BSD, find doesn't have the -printf predicate, stat can't output NULL bytes, and sed + awk can't handle NULL-delimited records.
Here's a solution that doesn't support newlines in paths but that safeguards against them by filtering them out:
#!/bin/bash
n=5
find . -maxdepth 1 -type f ! -path $'*\n*' -exec stat -f '%.9Fm %N' {} + |
sort -nrt ' ' -k1,1 |
awk -v n="$n" -F'^[^ ]* ' 'NR > n {printf "%s%c", $2, 0}' |
xargs -0 rm -f
note: I'm using bash because of the $'\n' notation. For sh you can define a variable containing a literal newline and use it instead.
Solution for UNIX & Linux (inspired from AIX/HP-UX/SunOS/BSD/Linux ls -b):
Some platforms don't provide find -printf, nor stat, nor support NUL-delimited records with stat/sort/awk/sed/xargs. That's why using perl is probably the most portable way to tackle the problem, because it is available by default in almost every OS.
I could have written the whole thing in perl but I didn't. I only use it for substituting stat and for encoding-decoding-escaping the filenames. The core logic is the same as the previous solutions and is implemented with POSIX tools.
note: perl's default stat has a resolution of a second, but starting from perl-5.8.9 you can get sub-second resolution with the stat function of the module Time::HiRes (when both the OS and the filesystem support it). That's what I'm using here; if your perl doesn't provide it then you can remove the ‑MTime::HiRes=stat from the command line.
n=5
find . '(' -name '.' -o -prune ')' -type f -exec \
perl -MTime::HiRes=stat -le '
foreach (#ARGV) {
#st = stat($_);
if ( #st > 0 ) {
s/([\\\n])/sprintf( "\\%03o", ord($1) )/ge;
print sprintf( "%.9f %s", $st[9], $_ );
}
else { print STDERR "stat: $_: $!"; }
}
' {} + |
sort -nrt ' ' -k1,1 |
sed -e "1,${n}d" -e 's/[^ ]* //' |
perl -l -ne '
s/\\([0-7]{3})/chr(oct($1))/ge;
s/(["\n])/"\\$1"/g;
print "\"$_\"";
' |
xargs -E '' sh -c '[ "$#" -gt 0 ] && rm -f "$#"' sh
Explanations:
For each file found, the first perl gets the modification time and outputs it along the encoded filename (each newline and backslash characters are replaced with the literals \012 and \134 respectively).
Now each time filename is guaranteed to be single-line, so POSIX sort and sed can safely work with this stream.
The second perl decodes the filenames and escapes them for POSIX xargs.
Lastly, xargs calls rm for deleting the files. The sh command is a trick that prevents xargs from running rm when there's no files to delete.
I realize this is an old thread, but maybe someone will benefit from this. This command will find files in the current directory :
for F in $(find . -maxdepth 1 -type f -name "*_srv_logs_*.tar.gz" -printf '%T# %p\n' | sort -r -z -n | tail -n+5 | awk '{ print $2; }'); do rm $F; done
This is a little more robust than some of the previous answers as it allows to limit your search domain to files matching expressions. First, find files matching whatever conditions you want. Print those files with the timestamps next to them.
find . -maxdepth 1 -type f -name "*_srv_logs_*.tar.gz" -printf '%T# %p\n'
Next, sort them by the timestamps:
sort -r -z -n
Then, knock off the 4 most recent files from the list:
tail -n+5
Grab the 2nd column (the filename, not the timestamp):
awk '{ print $2; }'
And then wrap that whole thing up into a for statement:
for F in $(); do rm $F; done
This may be a more verbose command, but I had much better luck being able to target conditional files and execute more complex commands against them.
If the filenames don't have spaces, this will work:
ls -C1 -t| awk 'NR>5'|xargs rm
If the filenames do have spaces, something like
ls -C1 -t | awk 'NR>5' | sed -e "s/^/rm '/" -e "s/$/'/" | sh
Basic logic:
get a listing of the files in time order, one column
get all but the first 5 (n=5 for this example)
first version: send those to rm
second version: gen a script that will remove them properly
With zsh
Assuming you don't care about present directories and you will not have more than 999 files (choose a bigger number if you want, or create a while loop).
[ 6 -le `ls *(.)|wc -l` ] && rm *(.om[6,999])
In *(.om[6,999]), the . means files, the o means sort order up, the m means by date of modification (put a for access time or c for inode change), the [6,999] chooses a range of file, so doesn't rm the 5 first.
Adaptation of #mklement0's excellent answer with some parameters and without needing to navigate to the folder containing the files to be deleted...
TARGET_FOLDER="/my/folder/path"
FILES_KEEP=5
ls -tp "$TARGET_FOLDER"**/* | grep -v '/$' | tail -n +$((FILES_KEEP+1)) | xargs -d '\n' -r rm --
[Ref(s).: https://stackoverflow.com/a/3572628/3223785 ]
Thanks! 😉
found interesting cmd in Sed-Onliners - Delete last 3 lines - fnd it perfect for another way to skin the cat (okay not) but idea:
#!/bin/bash
# sed cmd chng #2 to value file wish to retain
cd /opt/depot
ls -1 MyMintFiles*.zip > BigList
sed -n -e :a -e '1,2!{P;N;D;};N;ba' BigList > DeList
for i in `cat DeList`
do
echo "Deleted $i"
rm -f $i
#echo "File(s) gonzo "
#read junk
done
exit 0
Removes all but the 10 latest (most recents) files
ls -t1 | head -n $(echo $(ls -1 | wc -l) - 10 | bc) | xargs rm
If less than 10 files no file is removed and you will have :
error head: illegal line count -- 0
To count files with bash
I needed an elegant solution for the busybox (router), all xargs or array solutions were useless to me - no such command available there. find and mtime is not the proper answer as we are talking about 10 items and not necessarily 10 days. Espo's answer was the shortest and cleanest and likely the most unversal one.
Error with spaces and when no files are to be deleted are both simply solved the standard way:
rm "$(ls -td *.tar | awk 'NR>7')" 2>&-
Bit more educational version: We can do it all if we use awk differently. Normally, I use this method to pass (return) variables from the awk to the sh. As we read all the time that can not be done, I beg to differ: here is the method.
Example for .tar files with no problem regarding the spaces in the filename. To test, replace "rm" with the "ls".
eval $(ls -td *.tar | awk 'NR>7 { print "rm \"" $0 "\""}')
Explanation:
ls -td *.tar lists all .tar files sorted by the time. To apply to all the files in the current folder, remove the "d *.tar" part
awk 'NR>7... skips the first 7 lines
print "rm \"" $0 "\"" constructs a line: rm "file name"
eval executes it
Since we are using rm, I would not use the above command in a script! Wiser usage is:
(cd /FolderToDeleteWithin && eval $(ls -td *.tar | awk 'NR>7 { print "rm \"" $0 "\""}'))
In the case of using ls -t command will not do any harm on such silly examples as: touch 'foo " bar' and touch 'hello * world'. Not that we ever create files with such names in real life!
Sidenote. If we wanted to pass a variable to the sh this way, we would simply modify the print (simple form, no spaces tolerated):
print "VarName="$1
to set the variable VarName to the value of $1. Multiple variables can be created in one go. This VarName becomes a normal sh variable and can be normally used in a script or shell afterwards. So, to create variables with awk and give them back to the shell:
eval $(ls -td *.tar | awk 'NR>7 { print "VarName=\""$1"\"" }'); echo "$VarName"
leaveCount=5
fileCount=$(ls -1 *.log | wc -l)
tailCount=$((fileCount - leaveCount))
# avoid negative tail argument
[[ $tailCount < 0 ]] && tailCount=0
ls -t *.log | tail -$tailCount | xargs rm -f
I made this into a bash shell script. Usage: keep NUM DIR where NUM is the number of files to keep and DIR is the directory to scrub.
#!/bin/bash
# Keep last N files by date.
# Usage: keep NUMBER DIRECTORY
echo ""
if [ $# -lt 2 ]; then
echo "Usage: $0 NUMFILES DIR"
echo "Keep last N newest files."
exit 1
fi
if [ ! -e $2 ]; then
echo "ERROR: directory '$1' does not exist"
exit 1
fi
if [ ! -d $2 ]; then
echo "ERROR: '$1' is not a directory"
exit 1
fi
pushd $2 > /dev/null
ls -tp | grep -v '/' | tail -n +"$1" | xargs -I {} rm -- {}
popd > /dev/null
echo "Done. Kept $1 most recent files in $2."
ls $2|wc -l
Modified version of the answer of #Fabien if you want to specify a path. Useful if you're running the script elsewhere.
ls -tr /path/foo/ | head -n -5 | xargs -I% --no-run-if-empty rm /path/foo/%