I am trying to get files older than 3 days (by modified date, mtime) by find command (i want to exec rm command in future, doesnt matter why :D).
Problem is I am not root user and there are mix of the files, of course.
It looks, like simple script as hell gonna to stuck when occurs first permission denied message.
#!/bin/sh
find /tmp* -type f -mtime +3 -group jenkins -print
# -exec rm {} \;
Output is just error message (owned by root user)
find: ‘/tmp/systemd-private-dbe33161924b4616a037608d052f48d0-httpd.service-sFr4Wd’: Permission denied
Notice- there are many files which should occur in find command:
ll /tmp | less
....
drwxr-xr-x 2 jenkins jenkins 4096 Dec 11 02:33 02a47e28-4254-45d4-b69d-ed33b9ef3bad
drwxr-xr-x 2 jenkins jenkins 4096 Dec 6 15:10 03a1acc7-3040-430a-b5c3-9ce646407b93
drwxr-xr-x 2 jenkins jenkins 4096 Dec 5 12:27 062eb875-3216-4b2a-bae2-66106b66b0cd
drwxr-xr-x 2 jenkins jenkins 4096 Dec 9 06:51 0a6f8afc-0976-441b-980b-d0502f3903b1
drwxr-xr-x 2 jenkins jenkins 4096 Dec 5 13:07 1512475627515-0
drwxr-xr-x 2 jenkins jenkins 4096 Dec 5 13:07 1512475627836-0
drwxr-xr-x 2 jenkins jenkins 4096 Dec 5 13:07 1512475629807-0
drwxr-xr-x 2 jenkins jenkins 4096 Dec 5 13:07 1512475629829-0
drwxr-xr-x 2 jenkins jenkins 4096 Dec 5 13:07 1512475629843-0
.....
-rw-rw-r-- 1 root root 301 Dec 5 14:37 informix_disc.log
-rw-rw-r-- 1 root root 714 Dec 5 14:37 oracle_disc.log
-rw-rw-r-- 1 root root 602 Dec 5 14:37 sybase_disc.log
drwx------ 3 root root 4096 Nov 20 13:47 systemd-private-dbe33161924b4616a037608d052f48d0-httpd.service-sFr4Wd
drwxr-xr-x 2 root root 4096 Oct 30 09:39 vmware-config0
drwxr-xr-x 2 root root 4096 Nov 17 08:51 vmware-config1
...
I have tried to avoid permission denied message following:
1. 2>/dev/null OR 2>&- (add on the end, error output to null redirect)
2. find -type d ! -readable -prune -o ... (exclude not readable folders)
3. find /tmp* -type f -mtime +3 -group jenkins -print 2>&1 | grep -v "Permission denied" (should be same as above)
4. find -user jenkins, as well as -user $(whoami)
5. -perm -u+rwx
When I try to redirect error output to /dev/null, command output is blank
Kindly please,
am I doing something wrong, or what is the reason it gots stucked after first permission denied message?
Is there any better simple way how to do what I want?
IS any way how to ignore not permitted folders and make it working?
Thanks
Oh, problem was probably in logic finding files, I have made two commands (find directories and files separately)
#!/bin/sh
find /tmp -user $(whoami) -type d -mtime +3 -exec rm -rf {} \;
find /tmp -user $(whoami) -type f -mtime +3 -exec rm -f {} \;
And It looks like it works well, permission denied message still occurs, but commands are working.
Related
This question already has answers here:
How to ignore xargs commands if stdin input is empty?
(7 answers)
Closed 1 year ago.
me#new840:~/.config$ ls -al
total 196
drwx------ 40 me me 4096 Aug 22 22:08 .
drwxr-xr-x 52 me me 4096 Aug 22 12:12 ..
drwxr-xr-x 2 me me 4096 Jul 23 04:36 autostart
drwxr-xr-x 2 me me 4096 Jul 25 00:30 bcompare
drwx------ 20 me me 4096 Aug 22 22:37 Code
drwx------ 2 me me 4096 Aug 22 22:50 dconf
drwxr-xr-x 3 me me 4096 Aug 15 07:19 deadbeef
-rw------- 1 me me 1131 Nov 21 2016 dleyna-server-service.conf
me#new840:~/.config$ find /home/me/.config ! -user "me" -type f -print0 | xargs -0 ls -Al
total 188
drwxr-xr-x 2 me me 4096 Jul 23 04:36 autostart
drwxr-xr-x 2 me me 4096 Jul 25 00:30 bcompare
drwx------ 20 me me 4096 Aug 22 22:37 Code
drwx------ 2 me me 4096 Aug 22 22:56 dconf
drwxr-xr-x 3 me me 4096 Aug 15 07:19 deadbeef
-rw------- 1 me me 1131 Nov 21 2016 dleyna-server-service.conf
Actually,the result of find /home/me/.config ! -user "me" -type f -print0 | xargs -0 ls -Al should be empty.
How to avoid ls . when find | xargs ls has no result?
GNU xargs has the -r option to avoid doing anything if it receives no input, but this is not portable.
With find, the solution is simple: use -exec instead of xargs.
find /home/me/.config ! -user "me" -type f -exec ls -Al {} +
Tangentially, don't use ls in scripts.
Xargs supports the options -r or --no-run-if-empty. Then the ls command is not run when nothing is found.
Note that find supports the option -ls. Depending on what you want to see, you can skip xargs entirely.
Note: Assuming GNU environment.
I prefer to use bash script,because getfacl,setfacl has big problem(bug).
Before I ask this question,I also search a way of pure bash script to backup and restore owner and permission.It's a pity that no one acknowledged perfect answer.
Then I use a easy way to backup and restore owner and permission:
getfacl -R . >permissions.facl
setfacl --restore=permissions.facl
If I want to exclude .git from .,how to do?
Yes you can.
For example, my directory looks like
[root#967dd7743677 test]# ls -la
total 20
drwxr-xr-x 4 root root 4096 Jan 11 06:37 .
drwxr-xr-x 1 root root 4096 Jan 2 11:08 ..
drwxr-xr-x 8 root root 4096 Jan 2 12:56 .git
drwxr-xr-x 2 root root 4096 Jan 11 06:37 1one
-rw-r--r-- 1 root root 19 Jan 2 12:55 testfile
[root#967dd7743677 test]#
by using find you can exclude any directory you want
find . -type f -not -path '*/\.git/*' -exec getfacl -R {} \;
so via -exec we are calling getfacl -R.
and you can redirect the output
find . -type f -not -path '*/\.git/*' -exec getfacl -R {} \; > permissions.facl
Hope it helps.
This question already has answers here:
Unix find: multiple file types
(7 answers)
Closed 9 years ago.
in the below list i want to retrieve files starting with data and user. need to retrieve from subdirectories also.
drwxr-x--- 2 root root 4096 Jul 26 11:59 MIS
drwxr-x--- 2 root root 4096 Jul 26 12:04 recommendations
drwxr-x--- 3 root root 4096 Sep 11 07:45 script
drwxr-x--- 2 root root 4096 Sep 11 08:40 log
-rw-r----- 1 root root 0 Sep 11 09:48 data_collection.txt
-rw-r----- 1 root root 0 Sep 11 09:48 user_collection_12345.txt
Any help please?
This makes it:
find /your/dir -type f \( -name "data*" -o -name "user*" \)
^^^^^^^ ^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
files either data... or user...
I'm using bash shell.
Hi,ppl
Would be glad if someone could provide some kind of advice, because googling around yielded some answers
but couldn't still get the script to work.
I'am new to using bash script and got a script to modify because it was failing to copy
a large number of files from and input directory to an output directory after the files were processed.
Description:
We have a bunch of pdf's in a large directory.
We process a file called filename.pdf, after it's processed an additional file is created called filename.pdf.marker
Then both files filename.pdf.marker and filename.pdf shoud be moved from input/in directory to directory output/out.
We work with about 10 -15 thousands of files.
The script should do the following:
select all .marker file names
move.marker files from input/in directory to directory output/out (done in separate line)
remove the .marker from the selected filename,
move the file filename.pdf to the output/out directory
Old script (didn't work for a larger number of files) :
FILELIST=$(ls ${V04}/*.pdf.marker 2> /dev/null | sort)
for FILEMARKER in ${FILELIST}; do
FILENAME=${V04}/$(basename $FILEMARKER .marker)
mv ${FILENAME} ${VLOGDIR}/.
mv ${FILENAME}.marker ${VLOGDIR}/.
done
Because of that I needed to use xargs command.
Problem:
I managed to move the .marker files in a separate line.
Now i need to move the .pdf files with this script line.
find /input/in -iname "*.marker" -print0 | xargs -0 -r -I {} mv `basename {} .marker` /output/out
My problem lies in the part: `basename {} .marker`
Why isn't the string filename.pdf extracted from the string filename.pdf.marker, and substituted into the mv command ?
Any help i's welcome ;)
UPDATED
Corrected description of what script should do: Both filetypes .pdf
and .pdf.marker should be moved in my script not copied.
Added old script that didn't work well for larger amount of files.
The problem is that the command in backticks is executed once before xargs is ever invoked.
The fix is a bit harder, not least because your step 2 says 'copy' but the previous description suggests 'move'. I'd probably create a simple script to be invoked by xargs:
find /input/in -name '*.marker' -print0 | xargs -0 mover.sh
The contents of mover.sh might be:
for mrk_source in "$#"
do
pdf_source=$(echo "$mrk_source" | sed 's/\.marker$//')
mrk_target=$(echo "/output/out/$mrk_source" | sed 's%/input/in%%')
pdf_target=$(echo "/output/out/$pdf_source" | sed 's%/input/in%%')
mv "$mrk_source" "$mrk_target"
mv "$pdf_source" "$pdf_target"
done
Note that this code preserves any directory structure under /input/in but assumes that the corresponding directory exists under /output/out (without checking). It would be possible to alter the code to flatten any directory structure, or to create the directories as needed (exercise for the reader). There is a small sleight-of-hand going on in the file name manipulation in the two xxx_target assignment lines; I think it will work OK for relative names as well as absolute names, but be a little cautious with that part (test before using, in other words).
tripleee commented:
The echo and sed invocations are very brittle -- for example, echo on some platforms will interpret backslashes in the filename as escape sequences. Fortunately, you can use the shell's substitution mechanisms to mv "${mrk_source#.marker}" /output/out instead. (Why would you want to calculate the destination file name, when all you need to give to mv is the destination directory?)
I explained the destination file name - preserving sub-directories, so /input/in/dir1/abc.pdf goes to /output/out/dir1/abc.pdf; if you want to flatten the directory structure (or there is no directory structure), then simply specifying the destination is sufficient.
The problem with echo 'should not' be a problem in the sense that the original design of echo was simple and all the later additional ... baggage simply makes what should be utterly reliable into something horrendously unreliable. That said, there could be problems with names containing backticks, $(...) and so on. There are no problems with backticks or $(...) in the names. There is a problem with backslashes in the name.
$ mkdir -p input/in output/out
$ for name in a b 'c d' 'e f g' '$(cat x)' '`cat y`' 'a\\nb'
> do
> cp /dev/null input/in/"$name.pdf"
> cp /dev/null "input/in/$name.pdf.marker"
> done
$ ls -lR [io]*
input:
total 0
drwxr-xr-x 16 jleffler staff 544 Aug 22 00:45 in
input/in:
total 0
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\\nb.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\\nb.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf.marker
output:
total 0
drwxr-xr-x 2 jleffler staff 68 Aug 22 00:45 out
output/out:
$ find input/in -name '*.marker' -print0 | xargs -0 sh mover.sh
mv: rename input/in/a\nb.pdf to ./output/out/a
b.pdf: No such file or directory
$ ls -lR [io]*
input:
total 0
drwxr-xr-x 3 jleffler staff 102 Aug 22 00:46 in
input/in:
total 0
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\\nb.pdf
output:
total 0
drwxr-xr-x 15 jleffler staff 510 Aug 22 00:46 out
output/out:
total 0
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 $(cat x).pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 `cat y`.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 a\nb.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 b.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 c d.pdf.marker
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf
-rw-r--r-- 1 jleffler staff 0 Aug 22 00:45 e f g.pdf.marker
$
Using the Bash built-ins is sensible; I'm still stuck in the 1980s on occasion, and need reminding of that.
Solution that works with backslashes etc
for mrk_source in "$#"
do
pdf_source=${mrk_source%.marker}
mrk_target=${mrk_source/\/input\/in/\/output\/out}
pdf_target=${pdf_source/\/input\/in/\/output\/out}
mv "$mrk_source" "$mrk_target"
mv "$pdf_source" "$pdf_target"
done
With the same set of input files, this code works cleanly:
EDIT: As pointed out in the comments, this will not work if there are spaces in the filenames. In that case see #Jonathan Leffler's answer (even if there are no spaces now, you should probably use his version anyway, to avoid breakage when there suddenly are spaces...).
Since the command is expanded before it is executed, you can't use it that way. The command you'll give xargs would look like this:
xargs -0 -r -I {} mv {} /output/out
Since it tries to remove any path components, and the a .marker suffix, from the string {}.
I'd say you want to use a loop in this case:
for f in $(find /input/in -iname "*.marker"); do
mv `basename $f .marker` /output/out
done
With GNU Parallel you should be able to do:
ls "$V04"/*.pdf.marker | parallel -q mv {.} {} "$VLOGDIR"
This will work even if $V04 and $VLOGDIR contains ' " space \t.
Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
The problem with the old script was, that you try to catch all entries in one var, which have size limits.
You can solve that, if you must not sort the entries in this way:
ls -1 "${V04}/*.pdf,.marker | while read FM;
do
mv "${FM}" "${VLOGDIR}/"
mv "${V04}/$(basename "${FM}" .marker)" "${VLOGDIR}/"
done;
The backticks are executed at evaluation time, not when xargs runs. Perhaps try something like this?
find /input/in -iname "*.marker" -print0 |
xargs -r0 -i sh -c 'mv `basename "{}" .marker` /output/out; mv "{}" /output/out'
Edit: The shell is still problematic here; if the file name contains double quotes, it will not parse correctly. Using a separate script might be better:
find /input/in -iname "*.marker" -exec ./myscript {} \;
where myscript contains the simple moving commands:
#!/bin/sh
mv `basename "$1" .marker` /output/out
mv "$1" /output/out
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Delete all but the most recent X files in bash
I have a script to create incremental backups daily and I need to delete all backups but last 5.
For example, I have this folders:
drwxr-xr-x 4 root root 4096 Oct 29 01:10 2010-10-29
drwxr-xr-x 4 root root 4096 Oct 30 01:10 2010-10-30
drwxr-xr-x 4 root root 4096 Oct 31 01:10 2010-10-31
drwxr-xr-x 4 root root 4096 Nov 1 01:10 2010-11-01
drwxr-xr-x 4 root root 4096 Nov 2 01:10 2010-11-02
drwxr-xr-x 4 root root 4096 Nov 3 01:10 2010-11-03
drwxr-xr-x 4 root root 4096 Nov 4 01:10 2010-11-04
drwxr-xr-x 4 root root 4096 Nov 5 01:10 2010-11-05
drwxr-xr-x 4 root root 4096 Nov 6 01:10 2010-11-06
drwxr-xr-x 4 root root 4096 Nov 7 01:10 2010-11-07
drwxr-xr-x 4 root root 4096 Nov 8 01:10 2010-11-08
And I need to maintain only the last 5 directories and delete the others. After command execute, I need to have only this:
drwxr-xr-x 4 root root 4096 Nov 4 01:10 2010-11-04
drwxr-xr-x 4 root root 4096 Nov 5 01:10 2010-11-05
drwxr-xr-x 4 root root 4096 Nov 6 01:10 2010-11-06
drwxr-xr-x 4 root root 4096 Nov 7 01:10 2010-11-07
drwxr-xr-x 4 root root 4096 Nov 8 01:10 2010-11-08
I don't need to delete previous to 5 days, I need to delete all except 5 last directories :)
Now I'm using:
find /backup/increment -maxdepth 1 -type d -mtime +5 -exec rm -rf {} \;
But I need to improved not based in time :)
EDIT: This is an example for a server that do backups all days, but I need an script that delete all folders previous to last 5 because my computer do backups at 00:10 at night, but not all nights the backup is done it, because my computer isn't working all days, and I need to have always the last 5 backups :)
use the tail command to print lines starting with the n th line (Option -n +N):
rm `ls -t | tail -n +6`
ls -t outputs the current directory sorted by time. tail -n +6 takes al lines starting with the 6th line. Quoting with backticks feeds the result of the pipe into the rm command.
OLD SOLUTION, not correct ...
use the head command, which prints the first n lines of some output:
rm `ls -t1 | head -n 5`
ls -t outputs the current directory sorted by time. head -n 5 takes the first five entries of the previous output. Quoting with backticks feeds the result of the pipe into the rm command.
Please try out first before applying to live data :) ...
The first thing that came to my mind. It's not elegant:
a=0;
for i in `ls -t`;
do
a=`expr $a + 1`;
if [ $a -gt 5 ]; then
echo "removing $i";
rm -rf $i
fi;
done
create two dummy files with the start and the end date
touch -t 1010290000 before
touch -t 2011042359 after
find all the files between the 2 dummy files and "rm -rf" the result
find . -newer before \! -newer after -exec rm -rf {} \;
ls -tr | perl -ne '{#files = <>; print #files[0..$#files-5 ]}' | xargs -n1 echo rm -rf
You would remove the echo before the rm -rf to get it to work.
The trick will be the -t option to ls, which sorts by modification time, from newest to oldest.
A really naive solution, using a temporary file, might go this way:
ls -t > /tmp/file_list
num_files_to_keep=5
# wc -l gets the line count of a file
# first word of wc output is the actual line count, and that's all we need, so
# delete everything after the space.
num_files=`wc -l /tmp/file_list | sed "s/ .*//"`
#if appropriate you should add a check for num_files < num_files_to_keep
num_files_to_delete=$(( $num_files - $num_files_to_keep ))
rm `tail -n $num_files_to_delete /tmp/file_list`